2024-03-25T18:38:12.341 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T18:38:12.341 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --verbose --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621334 -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621334/orig.config.yaml 2024-03-25T18:38:12.383 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:38:12.461 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621334 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621334' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi081.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCN07iZlXHTVUpCzBERZtT0MJipiUFcqZGMF7B9u5DuDBQr05X96yQDfh6RwCfxsDrBc/m6JwKQt4dYEYGHw30c= smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAwY3yfX1UoEShtlLxQrA1uQ4Cv4BCM43c1vEnjfLjFwEVlcfA35mwkmImJeqzHFIeBGPMSMzTHJBoPWDpViQLY= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T18:38:12.462 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T18:38:12.462 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T18:38:12.462 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T18:38:12.464 INFO:teuthology.task.internal:Checking packages... 2024-03-25T18:38:12.487 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T18:38:12.488 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T18:38:12.488 INFO:teuthology.packaging:ref: None 2024-03-25T18:38:12.488 INFO:teuthology.packaging:tag: None 2024-03-25T18:38:12.488 INFO:teuthology.packaging:branch: main 2024-03-25T18:38:12.488 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:38:12.488 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T18:38:12.871 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T18:38:12.872 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T18:38:12.880 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T18:38:12.880 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T18:38:12.968 INFO:teuthology.task.internal:Saving configuration 2024-03-25T18:38:12.984 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T18:38:13.067 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:38:13.090 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621334', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:32:26.971765', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCN07iZlXHTVUpCzBERZtT0MJipiUFcqZGMF7B9u5DuDBQr05X96yQDfh6RwCfxsDrBc/m6JwKQt4dYEYGHw30c='} 2024-03-25T18:38:13.110 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621334', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:32:26.970161', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAwY3yfX1UoEShtlLxQrA1uQ4Cv4BCM43c1vEnjfLjFwEVlcfA35mwkmImJeqzHFIeBGPMSMzTHJBoPWDpViQLY='} 2024-03-25T18:38:13.110 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T18:38:13.158 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T18:38:13.158 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T18:38:13.158 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T18:38:13.275 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5224b570d0>, signals=[15]) 2024-03-25T18:38:13.275 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T18:38:13.281 INFO:teuthology.task.internal:Opening connections... 2024-03-25T18:38:13.281 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T18:38:13.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:38:13.373 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T18:38:13.374 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:38:13.463 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T18:38:13.469 DEBUG:teuthology.orchestra.run.smithi081:> uname -m 2024-03-25T18:38:13.486 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2024-03-25T18:38:13.486 DEBUG:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2024-03-25T18:38:13.542 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="CentOS Stream" 2024-03-25T18:38:13.543 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="9" 2024-03-25T18:38:13.543 INFO:teuthology.orchestra.run.smithi081.stdout:ID="centos" 2024-03-25T18:38:13.543 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:38:13.543 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="9" 2024-03-25T18:38:13.543 INFO:teuthology.orchestra.run.smithi081.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:38:13.543 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:38:13.543 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31" 2024-03-25T18:38:13.543 INFO:teuthology.orchestra.run.smithi081.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:38:13.543 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:38:13.543 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:38:13.543 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:38:13.543 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:38:13.543 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:38:13.544 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2024-03-25T18:38:13.569 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-03-25T18:38:13.586 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-03-25T18:38:13.586 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-03-25T18:38:13.643 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-03-25T18:38:13.643 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-03-25T18:38:13.643 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-03-25T18:38:13.643 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:38:13.643 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-03-25T18:38:13.643 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:38:13.643 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:38:13.643 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-03-25T18:38:13.643 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:38:13.643 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:38:13.644 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:38:13.644 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:38:13.644 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:38:13.644 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:38:13.644 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-03-25T18:38:13.669 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T18:38:13.677 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T18:38:13.728 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T18:38:13.729 DEBUG:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:38:13.731 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:38:13.747 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T18:38:13.763 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T18:38:13.763 DEBUG:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:38:13.788 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:38:13.827 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T18:38:13.885 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T18:38:13.900 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T18:38:13.900 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T18:38:13.900 DEBUG:teuthology.orchestra.run.smithi081:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:38:13.901 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:38:13.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:38:13.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:38:13.918 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-03-25T18:38:13.918 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-03-25T18:38:13.975 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:38:13.975 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:38:13.975 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-361.el9.x86_64 2024-03-25T18:38:13.975 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-03-25T18:38:13.976 INFO:teuthology.task.kernel:Running kernel on smithi081: 5.14.0-361.el9.x86_64 2024-03-25T18:38:13.976 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum install -y kernel 2024-03-25T18:38:16.224 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-25T18:38:16.486 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - AppStream 8.2 MB/s | 6.7 MB 00:00 2024-03-25T18:38:17.875 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-25T18:38:18.112 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-25T18:38:24.485 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 1.4 MB/s | 8.0 MB 00:05 2024-03-25T18:38:24.937 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 1.2 MB/s | 8.0 MB 00:06 2024-03-25T18:38:31.961 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 2.9 MB/s | 19 MB 00:06 2024-03-25T18:38:32.467 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 2.9 MB/s | 19 MB 00:06 2024-03-25T18:38:40.653 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 1.2 MB/s | 6.1 MB 00:05 2024-03-25T18:38:40.843 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 1.2 MB/s | 6.1 MB 00:04 2024-03-25T18:38:42.269 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-25T18:38:42.539 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-03-25T18:38:43.917 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-25T18:38:44.844 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 8.4 MB/s | 21 MB 00:02 2024-03-25T18:38:48.859 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-25T18:38:49.827 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-03-25T18:38:51.040 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:38:51.041 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:38:51.127 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:38:51.133 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:38:51.133 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-03-25T18:38:51.133 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:38:51.133 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:38:51.133 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:38:51.134 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T18:38:51.134 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:38:51.134 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:38:51.134 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:38:51.134 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:38:51.134 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:38:51.134 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:38:51.134 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-03-25T18:38:51.134 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:38:51.135 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 76 M 2024-03-25T18:38:51.135 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 123 M 2024-03-25T18:38:51.135 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:38:51.356 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:38:51.357 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:38:51.357 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:38:51.504 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:38:51.520 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:38:51.542 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:38:51.550 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:38:51.556 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:38:51.567 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:38:51.625 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:38:51.637 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:38:51.647 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:38:51.776 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:38:51.790 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:38:51.820 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:38:51.952 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:38:51.965 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:38:51.967 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:38:51.967 INFO:teuthology.orchestra.run.smithi081.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:38:51.989 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:38:52.061 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:38:52.069 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:38:52.069 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-03-25T18:38:52.070 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:38:52.070 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:38:52.070 INFO:teuthology.orchestra.run.smithi081.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:38:52.070 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T18:38:52.070 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:38:52.070 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:38:52.070 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:38:52.070 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:38:52.070 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:38:52.070 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:38:52.070 INFO:teuthology.orchestra.run.smithi081.stdout:Install 4 Packages 2024-03-25T18:38:52.070 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:38:52.071 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 76 M 2024-03-25T18:38:52.071 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 123 M 2024-03-25T18:38:52.071 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:38:52.216 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.218 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.226 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.237 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.237 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.246 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.316 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:38:52.318 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:38:52.321 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:38:52.327 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:38:52.346 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.351 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:38:52.380 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:38:52.387 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.420 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.421 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.454 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.462 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:38:52.470 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:38:52.497 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:38:52.498 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:38:52.505 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:38:52.515 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:38:52.520 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.592 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:38:52.609 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:38:52.671 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:38:52.677 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:38:52.680 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:38:52.689 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:38:52.692 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:38:52.727 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:38:52.771 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.779 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:38:52.783 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.812 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.849 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.857 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.891 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:52.935 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 39 kB/s | 66 kB 00:01 2024-03-25T18:38:53.055 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:38:53.076 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:38:53.105 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:38:53.111 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:38:53.188 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:38:53.208 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:38:53.234 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:38:53.239 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:38:53.341 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:38:53.376 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:38:53.388 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:38:53.406 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:53.407 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:38:53.439 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:38:53.447 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:53.477 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:38:53.480 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:53.480 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:53.514 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:53.597 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:53.652 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:38:53.664 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:38:53.693 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:38:53.732 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:38:53.767 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:38:53.771 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:38:53.879 INFO:teuthology.orchestra.run.smithi081.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 38 kB/s | 66 kB 00:01 2024-03-25T18:38:53.893 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 13 MB/s | 33 MB 00:02 2024-03-25T18:38:53.896 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:53.949 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:54.044 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): kernel-core-5.14.0-432.el9.x86_64.rpm 5.3 MB/s | 15 MB 00:02 2024-03-25T18:38:54.045 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:38:54.064 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:54.145 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:38:54.246 INFO:teuthology.orchestra.run.smithi081.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 7.0 MB/s | 15 MB 00:02 2024-03-25T18:38:54.249 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:38:54.297 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:38:54.325 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:38:54.353 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:54.407 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:54.524 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:38:54.558 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:38:54.624 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:38:55.096 INFO:teuthology.orchestra.run.smithi081.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 11 MB/s | 33 MB 00:02 2024-03-25T18:38:55.227 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 12 MB/s | 27 MB 00:02 2024-03-25T18:38:55.232 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:38:55.232 INFO:teuthology.orchestra.run.smithi098.stdout:Total 19 MB/s | 76 MB 00:04 2024-03-25T18:38:55.261 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:38:55.282 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:38:55.413 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:38:55.577 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:38:55.898 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:38:56.179 INFO:teuthology.orchestra.run.smithi081.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 12 MB/s | 27 MB 00:02 2024-03-25T18:38:56.184 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:38:56.184 INFO:teuthology.orchestra.run.smithi081.stdout:Total 18 MB/s | 76 MB 00:04 2024-03-25T18:38:56.255 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:38:56.255 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:38:56.847 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:38:57.205 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:38:57.206 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:38:57.248 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:38:57.248 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:38:58.202 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:38:58.202 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:38:59.722 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:39:00.327 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:39:00.403 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:39:00.655 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:39:01.263 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:39:01.357 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:39:03.133 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:39:03.256 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:39:04.060 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:39:04.202 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:39:07.980 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:39:08.015 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:39:08.898 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:39:08.935 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:39:12.742 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:39:13.658 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:39:42.230 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:39:42.230 INFO:teuthology.orchestra.run.smithi098.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:39:42.230 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:39:42.275 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:39:42.597 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:39:42.597 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:39:42.597 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:39:42.597 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:39:43.003 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:39:43.003 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:39:43.003 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:39:43.003 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:39:43.003 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:39:43.003 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:39:43.003 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:39:43.003 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:39:43.003 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:39:43.145 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:39:43.145 INFO:teuthology.orchestra.run.smithi081.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:39:43.145 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:39:43.190 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:39:43.351 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:39:43.442 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:39:43.442 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:39:43.443 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:39:43.443 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:39:43.443 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:39:43.443 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T18:39:43.443 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:39:43.443 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:39:43.522 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:39:43.523 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:39:43.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2024-03-25T18:39:43.557 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:39:43.557 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:39:43.557 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:39:43.557 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:39:43.565 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2024-03-25T18:39:43.566 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2024-03-25T18:39:43.566 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.06 2024-03-25T18:39:43.566 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 68.el9 2024-03-25T18:39:43.566 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2024-03-25T18:39:43.566 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:39:43.566 INFO:teuthology.orchestra.run.smithi098.stdout:Group : Unspecified 2024-03-25T18:39:43.566 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 8264254 2024-03-25T18:39:43.566 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2024-03-25T18:39:43.566 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:39:43.566 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:39:43.566 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:39:43.566 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:39:43.566 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : builder@centos.org 2024-03-25T18:39:43.566 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2024-03-25T18:39:43.567 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:39:43.567 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2024-03-25T18:39:43.567 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2024-03-25T18:39:43.567 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:39:43.567 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:39:43.567 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:39:43.567 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:39:43.567 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2024-03-25T18:39:43.567 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:39:43.567 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:39:43.568 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:39:43.568 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:39:43.568 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:39:43.953 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:39:43.953 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:39:43.953 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:39:43.953 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:39:43.953 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:39:43.953 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:39:43.953 INFO:teuthology.orchestra.run.smithi081.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:39:43.954 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:39:43.954 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:39:44.081 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-03-25T18:39:44.273 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:39:44.381 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:39:44.381 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:39:44.382 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:39:44.382 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:39:44.382 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:39:44.382 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T18:39:44.382 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi081.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:39:44.382 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:39:44.461 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:39:44.462 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:39:44.462 DEBUG:teuthology.orchestra.run.smithi081:> sudo rpm -qi grub2-tools 2024-03-25T18:39:44.496 INFO:teuthology.orchestra.run.smithi081.stdout:Name : grub2-tools 2024-03-25T18:39:44.496 INFO:teuthology.orchestra.run.smithi081.stdout:Epoch : 1 2024-03-25T18:39:44.496 INFO:teuthology.orchestra.run.smithi081.stdout:Version : 2.06 2024-03-25T18:39:44.496 INFO:teuthology.orchestra.run.smithi081.stdout:Release : 68.el9 2024-03-25T18:39:44.496 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: x86_64 2024-03-25T18:39:44.496 INFO:teuthology.orchestra.run.smithi081.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:39:44.496 INFO:teuthology.orchestra.run.smithi081.stdout:Group : Unspecified 2024-03-25T18:39:44.497 INFO:teuthology.orchestra.run.smithi081.stdout:Size : 8264254 2024-03-25T18:39:44.497 INFO:teuthology.orchestra.run.smithi081.stdout:License : GPLv3+ 2024-03-25T18:39:44.497 INFO:teuthology.orchestra.run.smithi081.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:39:44.497 INFO:teuthology.orchestra.run.smithi081.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:39:44.497 INFO:teuthology.orchestra.run.smithi081.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:39:44.497 INFO:teuthology.orchestra.run.smithi081.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:39:44.497 INFO:teuthology.orchestra.run.smithi081.stdout:Packager : builder@centos.org 2024-03-25T18:39:44.497 INFO:teuthology.orchestra.run.smithi081.stdout:Vendor : CentOS 2024-03-25T18:39:44.497 INFO:teuthology.orchestra.run.smithi081.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:39:44.497 INFO:teuthology.orchestra.run.smithi081.stdout:Summary : Support tools for GRUB. 2024-03-25T18:39:44.497 INFO:teuthology.orchestra.run.smithi081.stdout:Description : 2024-03-25T18:39:44.497 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:39:44.497 INFO:teuthology.orchestra.run.smithi081.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:39:44.497 INFO:teuthology.orchestra.run.smithi081.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:39:44.498 INFO:teuthology.orchestra.run.smithi081.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:39:44.498 INFO:teuthology.orchestra.run.smithi081.stdout:hardware devices. 2024-03-25T18:39:44.498 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:39:44.498 INFO:teuthology.orchestra.run.smithi081.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:39:44.499 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:39:44.499 INFO:teuthology.task.kernel:Updating grub on smithi081 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:39:44.499 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:39:45.142 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2024-03-25T18:39:45.519 INFO:teuthology.orchestra.run.smithi098.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:39:45.552 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-03-25T18:39:45.555 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-03-25T18:39:45.570 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.WC2j8gDRKp 2024-03-25T18:39:45.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WC2j8gDRKp 2024-03-25T18:39:45.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.WC2j8gDRKp 2024-03-25T18:39:45.772 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.WC2j8gDRKp is 6KB 2024-03-25T18:39:45.820 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.WC2j8gDRKp 2024-03-25T18:39:45.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:39:45.898 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:39:45.899 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:39:45.899 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:39:45.899 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:39:45.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:39:46.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-03-25T18:39:46.620 INFO:teuthology.orchestra.run.smithi081.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:39:46.652 INFO:teuthology.orchestra.run.smithi081.stderr:done 2024-03-25T18:39:46.655 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-03-25T18:39:46.670 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.JaRb8xlRJy 2024-03-25T18:39:46.670 DEBUG:teuthology.orchestra.run.smithi081:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.JaRb8xlRJy 2024-03-25T18:39:46.738 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.JaRb8xlRJy 2024-03-25T18:39:46.874 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.JaRb8xlRJy is 6KB 2024-03-25T18:39:46.893 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.JaRb8xlRJy 2024-03-25T18:39:46.907 DEBUG:teuthology.orchestra.run.smithi081:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:39:46.975 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:39:46.975 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:39:46.975 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:39:46.975 INFO:teuthology.orchestra.run.smithi081.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:39:46.977 DEBUG:teuthology.orchestra.run.smithi081:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:39:47.135 DEBUG:teuthology.orchestra.run.smithi081:> sudo shutdown -r now 2024-03-25T18:40:16.086 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T18:40:16.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-25T18:40:16.088 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:40:17.138 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T18:40:17.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-03-25T18:40:17.139 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:40:34.595 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-25T18:40:35.614 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.81 2024-03-25T18:40:43.601 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-25T18:40:43.602 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:40:44.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-03-25T18:40:44.616 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:40:46.654 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-25T18:40:58.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-25T18:40:58.667 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:41:30.554 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-25T18:41:31.113 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-25T18:41:31.113 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:41:31.113 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-03-25T18:41:31.127 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:41:31.128 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:41:31.128 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:41:31.128 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:41:32.128 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T18:41:32.129 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:41:32.204 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-03-25T18:41:32.241 DEBUG:teuthology.parallel:result is None 2024-03-25T18:41:44.619 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T18:41:56.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-03-25T18:41:56.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:41:57.080 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-03-25T18:41:57.459 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-03-25T18:41:57.459 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:41:57.459 DEBUG:teuthology.orchestra.run.smithi081:> uname -r 2024-03-25T18:41:57.475 INFO:teuthology.orchestra.run.smithi081.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:41:57.475 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:41:57.475 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:41:57.475 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:41:58.476 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T18:41:58.476 DEBUG:teuthology.orchestra.run.smithi081:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:41:58.558 INFO:teuthology.orchestra.run.smithi081.stdout:ttyS1 2024-03-25T18:41:58.590 DEBUG:teuthology.parallel:result is None 2024-03-25T18:41:58.590 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T18:41:58.599 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T18:41:58.599 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:41:58.602 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:41:58.620 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T18:41:58.628 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T18:41:58.641 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T18:41:58.642 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:41:58.658 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:41:58.731 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T18:41:58.739 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T18:41:58.740 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:41:58.742 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:41:58.776 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:41:58.781 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:41:58.790 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:41:58.795 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:41:58.797 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T18:41:58.812 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T18:41:58.813 DEBUG:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:41:58.839 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:41:58.869 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T18:41:58.879 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T18:41:58.879 DEBUG:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:41:58.908 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:41:58.926 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:41:59.004 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:41:59.054 DEBUG:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:41:59.137 DEBUG:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:41:59.179 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:41:59.179 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:41:59.246 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:41:59.291 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:41:59.340 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:41:59.424 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:41:59.466 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:41:59.466 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:41:59.533 DEBUG:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2024-03-25T18:41:59.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-03-25T18:41:59.580 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:41:59.635 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:42:00.133 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T18:42:00.140 INFO:teuthology.task.internal:Starting timer... 2024-03-25T18:42:00.140 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T18:42:00.153 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T18:42:00.163 DEBUG:teuthology.orchestra.run.smithi081:> sudo service auditd rotate 2024-03-25T18:42:00.210 INFO:teuthology.orchestra.run.smithi081.stdout:Rotating logs: 2024-03-25T18:42:00.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-03-25T18:42:00.270 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-03-25T18:42:00.272 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T18:42:00.272 DEBUG:teuthology.orchestra.run.smithi081:> /usr/sbin/getenforce 2024-03-25T18:42:00.303 INFO:teuthology.orchestra.run.smithi081.stdout:Permissive 2024-03-25T18:42:00.304 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-03-25T18:42:00.334 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-03-25T18:42:00.334 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi081.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi098.front.sepia.ceph.com': 'permissive'} 2024-03-25T18:42:00.334 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:42:00.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:42:00.372 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:42:00.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:42:00.401 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T18:42:00.401 DEBUG:teuthology.orchestra.run.smithi081:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:42:00.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:42:00.510 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T18:42:00.520 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T18:42:00.539 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T18:42:00.574 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi081.front.sepia.ceph.com,smithi098.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T18:46:13.585 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi081.front.sepia.ceph.com'), Remote(name='ubuntu@smithi098.front.sepia.ceph.com')] 2024-03-25T18:46:13.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-03-25T18:46:13.588 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi081.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:46:13.679 DEBUG:teuthology.orchestra.run.smithi081:> true 2024-03-25T18:46:13.765 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi081.front.sepia.ceph.com' 2024-03-25T18:46:13.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-25T18:46:13.766 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:46:13.855 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-25T18:46:13.941 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-25T18:46:13.941 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T18:46:13.953 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T18:46:13.953 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:46:13.953 DEBUG:teuthology.orchestra.run.smithi081:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:46:13.956 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:46:13.956 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:46:13.985 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:46:14.002 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:46:14.012 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:46:14.029 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:46:14.030 INFO:teuthology.orchestra.run.smithi081.stderr:sudo: ntpd: command not found 2024-03-25T18:46:14.044 INFO:teuthology.orchestra.run.smithi081.stdout:506 Cannot talk to daemon 2024-03-25T18:46:14.058 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-03-25T18:46:14.060 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:46:14.072 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-03-25T18:46:14.077 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:46:14.088 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:46:14.104 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:46:14.135 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:46:14.139 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:46:14.139 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-03-25T18:46:14.139 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:46:14.140 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:46:14.140 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:46:14.140 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:46:14.157 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:46:14.160 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:46:14.160 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-25T18:46:14.160 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:46:14.160 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:46:14.160 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:46:14.160 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:46:14.161 INFO:teuthology.run_tasks:Running task install... 2024-03-25T18:46:14.170 DEBUG:teuthology.task.install:project ceph 2024-03-25T18:46:14.170 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T18:46:14.170 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T18:46:14.170 INFO:teuthology.task.install:Using flavor: default 2024-03-25T18:46:14.176 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T18:46:14.176 INFO:teuthology.task.install:extra packages: [] 2024-03-25T18:46:14.176 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:46:14.177 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:46:14.179 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:46:14.179 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:46:14.324 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:46:14.324 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:46:14.389 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:46:14.389 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:46:14.727 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:46:14.728 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T18:46:14.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:46:14.735 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:46:14.735 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T18:46:14.735 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:46:14.756 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:46:14.756 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:46:14.764 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:46:14.764 DEBUG:teuthology.orchestra.run.smithi081:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:46:14.824 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:46:14.833 DEBUG:teuthology.orchestra.run.smithi081:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:46:14.904 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:46:14.913 DEBUG:teuthology.orchestra.run.smithi081:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:46:14.971 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-03-25T18:46:14.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-03-25T18:46:14.979 INFO:teuthology.orchestra.run.smithi081.stdout:check_obsoletes = 1 2024-03-25T18:46:14.980 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum clean all 2024-03-25T18:46:15.216 INFO:teuthology.orchestra.run.smithi098.stdout:74 files removed 2024-03-25T18:46:15.235 INFO:teuthology.orchestra.run.smithi081.stdout:74 files removed 2024-03-25T18:46:15.245 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw 2024-03-25T18:46:15.262 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-radosgw 2024-03-25T18:46:16.632 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 75 kB/s | 82 kB 00:01 2024-03-25T18:46:16.783 INFO:teuthology.orchestra.run.smithi081.stdout:ceph packages for x86_64 67 kB/s | 82 kB 00:01 2024-03-25T18:46:17.515 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T18:46:17.692 INFO:teuthology.orchestra.run.smithi081.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T18:46:18.441 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 2.1 kB/s | 1.9 kB 00:00 2024-03-25T18:46:18.501 INFO:teuthology.orchestra.run.smithi081.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-03-25T18:46:19.110 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T18:46:19.120 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T18:46:20.702 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-03-25T18:46:20.745 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-25T18:46:25.023 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:03 2024-03-25T18:46:25.410 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 8.0 MB 00:04 2024-03-25T18:46:30.440 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 19 MB 00:04 2024-03-25T18:46:30.870 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 19 MB 00:04 2024-03-25T18:46:37.729 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.1 MB 00:03 2024-03-25T18:46:38.005 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.1 MB 00:03 2024-03-25T18:46:39.985 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-25T18:46:40.794 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-25T18:46:42.080 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:01 2024-03-25T18:46:42.406 INFO:teuthology.orchestra.run.smithi081.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-25T18:46:46.995 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-03-25T18:46:47.337 INFO:teuthology.orchestra.run.smithi081.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-25T18:46:49.439 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:46:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:46:49.453 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T18:46:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:46:49.453 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:46:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:46:49.454 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-03-25T18:46:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:46:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:46:49.454 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T18:46:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options 2024-03-25T18:46:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:46:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:46:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:46:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:46:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:46:49.454 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:46:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:46:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:46:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:46:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:46:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:46:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:46:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:46:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:46:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:46:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:46:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:46:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse 2024-03-25T18:46:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:46:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:46:49.455 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:46:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:46:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:46:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:46:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:46:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:46:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:46:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:46:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:46:49.456 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-03-25T18:46:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:46:49.456 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:46:49.456 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:46:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:46:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:Install 30 Packages 2024-03-25T18:46:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 2 Packages 2024-03-25T18:46:49.457 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:46:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 58 M 2024-03-25T18:46:49.457 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:46:49.754 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:46:49.766 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:46:49.766 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T18:46:49.766 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:46:49.766 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:46:49.766 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:46:49.766 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrading: 2024-03-25T18:46:49.766 INFO:teuthology.orchestra.run.smithi081.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:46:49.766 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:46:49.766 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T18:46:49.766 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options 2024-03-25T18:46:49.767 INFO:teuthology.orchestra.run.smithi081.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:46:49.767 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:46:49.767 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:46:49.767 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:46:49.767 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:46:49.767 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:46:49.767 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:46:49.767 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:46:49.767 INFO:teuthology.orchestra.run.smithi081.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:46:49.767 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:46:49.767 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:46:49.767 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:46:49.767 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:46:49.767 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:46:49.767 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:46:49.768 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:46:49.768 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:46:49.768 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse 2024-03-25T18:46:49.768 INFO:teuthology.orchestra.run.smithi081.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:46:49.768 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:46:49.768 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:46:49.768 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:46:49.768 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:46:49.768 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:46:49.768 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:46:49.768 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:46:49.768 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:46:49.768 INFO:teuthology.orchestra.run.smithi081.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:46:49.768 INFO:teuthology.orchestra.run.smithi081.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:46:49.769 INFO:teuthology.orchestra.run.smithi081.stdout:Installing weak dependencies: 2024-03-25T18:46:49.769 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:46:49.769 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:46:49.769 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:46:49.769 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:46:49.769 INFO:teuthology.orchestra.run.smithi081.stdout:Install 30 Packages 2024-03-25T18:46:49.769 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrade 2 Packages 2024-03-25T18:46:49.769 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:46:49.769 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 58 M 2024-03-25T18:46:49.770 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:47:03.645 INFO:teuthology.orchestra.run.smithi098.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 397 kB/s | 5.3 MB 00:13 2024-03-25T18:47:04.213 INFO:teuthology.orchestra.run.smithi098.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 44 kB/s | 25 kB 00:00 2024-03-25T18:47:06.153 INFO:teuthology.orchestra.run.smithi098.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 382 kB/s | 742 kB 00:01 2024-03-25T18:47:07.904 INFO:teuthology.orchestra.run.smithi098.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 285 kB/s | 498 kB 00:01 2024-03-25T18:47:09.679 INFO:teuthology.orchestra.run.smithi081.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 281 kB/s | 5.3 MB 00:19 2024-03-25T18:47:09.795 INFO:teuthology.orchestra.run.smithi081.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 216 kB/s | 25 kB 00:00 2024-03-25T18:47:12.021 INFO:teuthology.orchestra.run.smithi081.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 333 kB/s | 742 kB 00:02 2024-03-25T18:47:14.229 INFO:teuthology.orchestra.run.smithi081.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 226 kB/s | 498 kB 00:02 2024-03-25T18:47:16.937 INFO:teuthology.orchestra.run.smithi098.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 368 kB/s | 9.7 MB 00:27 2024-03-25T18:47:17.145 INFO:teuthology.orchestra.run.smithi098.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 217 kB/s | 45 kB 00:00 2024-03-25T18:47:17.554 INFO:teuthology.orchestra.run.smithi098.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 321 kB/s | 131 kB 00:00 2024-03-25T18:47:18.088 INFO:teuthology.orchestra.run.smithi098.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 303 kB/s | 162 kB 00:00 2024-03-25T18:47:19.063 INFO:teuthology.orchestra.run.smithi098.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 329 kB/s | 320 kB 00:00 2024-03-25T18:47:19.921 INFO:teuthology.orchestra.run.smithi098.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 347 kB/s | 298 kB 00:00 2024-03-25T18:47:20.330 INFO:teuthology.orchestra.run.smithi098.stdout:(11/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 420 kB/s | 5.1 MB 00:12 2024-03-25T18:47:20.355 INFO:teuthology.orchestra.run.smithi098.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 229 kB/s | 99 kB 00:00 2024-03-25T18:47:20.480 INFO:teuthology.orchestra.run.smithi098.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-03-25T18:47:20.505 INFO:teuthology.orchestra.run.smithi098.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 603 kB/s | 106 kB 00:00 2024-03-25T18:47:20.531 INFO:teuthology.orchestra.run.smithi098.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-25T18:47:20.581 INFO:teuthology.orchestra.run.smithi098.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-25T18:47:20.606 INFO:teuthology.orchestra.run.smithi098.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-25T18:47:20.632 INFO:teuthology.orchestra.run.smithi098.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2024-03-25T18:47:20.696 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:47:20.696 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:47:20.820 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:47:20.846 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:47:20.857 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:47:20.875 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:20.880 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:47:20.927 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:20.936 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:20.981 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:21.028 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:47:21.047 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:47:21.163 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:47:21.193 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:47:21.324 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:47:21.356 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:47:21.674 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:47:21.707 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:47:21.812 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:47:21.844 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:47:21.890 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:47:21.923 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:47:21.966 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:21.975 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:21.984 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:21.992 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:22.098 INFO:teuthology.orchestra.run.smithi098.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 24 kB/s | 36 kB 00:01 2024-03-25T18:47:22.269 INFO:teuthology.orchestra.run.smithi098.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 133 kB/s | 221 kB 00:01 2024-03-25T18:47:22.301 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:47:22.441 INFO:teuthology.orchestra.run.smithi098.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.8 MB/s | 309 kB 00:00 2024-03-25T18:47:22.441 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:47:22.477 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:47:22.494 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:22.547 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:22.613 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:47:22.699 INFO:teuthology.orchestra.run.smithi098.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-03-25T18:47:22.733 INFO:teuthology.orchestra.run.smithi081.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 306 kB/s | 9.7 MB 00:32 2024-03-25T18:47:22.799 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:47:22.833 INFO:teuthology.orchestra.run.smithi098.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 238 kB/s | 26 kB 00:00 2024-03-25T18:47:22.892 INFO:teuthology.orchestra.run.smithi098.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 831 kB/s | 49 kB 00:00 2024-03-25T18:47:22.892 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:47:22.950 INFO:teuthology.orchestra.run.smithi098.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-25T18:47:22.972 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:47:23.017 INFO:teuthology.orchestra.run.smithi098.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-25T18:47:23.101 INFO:teuthology.orchestra.run.smithi098.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.1 MB/s | 192 kB 00:00 2024-03-25T18:47:23.177 INFO:teuthology.orchestra.run.smithi098.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-03-25T18:47:23.178 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:47:23.272 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:47:23.325 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:23.342 INFO:teuthology.orchestra.run.smithi098.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:23.385 INFO:teuthology.orchestra.run.smithi098.stdout:(29/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 37 kB/s | 47 kB 00:01 2024-03-25T18:47:23.720 INFO:teuthology.orchestra.run.smithi081.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 46 kB/s | 45 kB 00:00 2024-03-25T18:47:24.688 INFO:teuthology.orchestra.run.smithi081.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 136 kB/s | 131 kB 00:00 2024-03-25T18:47:25.455 INFO:teuthology.orchestra.run.smithi081.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 211 kB/s | 162 kB 00:00 2024-03-25T18:47:26.812 INFO:teuthology.orchestra.run.smithi081.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 236 kB/s | 320 kB 00:01 2024-03-25T18:47:27.805 INFO:teuthology.orchestra.run.smithi081.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 300 kB/s | 298 kB 00:00 2024-03-25T18:47:28.113 INFO:teuthology.orchestra.run.smithi081.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 322 kB/s | 99 kB 00:00 2024-03-25T18:47:28.230 INFO:teuthology.orchestra.run.smithi081.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 912 kB/s | 106 kB 00:00 2024-03-25T18:47:28.281 INFO:teuthology.orchestra.run.smithi081.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T18:47:28.305 INFO:teuthology.orchestra.run.smithi081.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-25T18:47:28.364 INFO:teuthology.orchestra.run.smithi081.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T18:47:28.406 INFO:teuthology.orchestra.run.smithi081.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T18:47:28.431 INFO:teuthology.orchestra.run.smithi081.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2024-03-25T18:47:28.496 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:47:28.627 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:47:28.655 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:28.712 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:28.884 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:47:29.035 INFO:teuthology.orchestra.run.smithi098.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 590 kB/s | 3.4 MB 00:05 2024-03-25T18:47:29.290 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:47:29.355 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:47:29.553 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:47:29.588 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:47:29.749 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:47:29.838 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:47:29.894 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:29.912 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:30.115 INFO:teuthology.orchestra.run.smithi081.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 131 kB/s | 221 kB 00:01 2024-03-25T18:47:30.147 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:47:30.179 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:47:30.198 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:30.251 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:30.331 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:47:30.417 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:47:30.452 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:47:30.493 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:47:30.510 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:47:30.655 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:47:30.734 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:47:30.788 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:30.806 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:30.840 INFO:teuthology.orchestra.run.smithi081.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 49 kB/s | 36 kB 00:00 2024-03-25T18:47:30.872 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:47:30.905 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:47:30.923 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:30.988 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:31.068 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:47:31.149 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:47:31.187 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:47:31.224 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:47:31.241 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:47:31.386 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:47:31.507 INFO:teuthology.orchestra.run.smithi081.stdout:(20/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 302 kB/s | 5.1 MB 00:17 2024-03-25T18:47:31.508 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:47:31.570 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:31.594 INFO:teuthology.orchestra.run.smithi081.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:47:31.624 INFO:teuthology.orchestra.run.smithi081.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 60 kB/s | 47 kB 00:00 2024-03-25T18:47:31.732 INFO:teuthology.orchestra.run.smithi081.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.3 MB/s | 309 kB 00:00 2024-03-25T18:47:31.783 INFO:teuthology.orchestra.run.smithi081.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 532 kB/s | 26 kB 00:00 2024-03-25T18:47:31.833 INFO:teuthology.orchestra.run.smithi081.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 976 kB/s | 49 kB 00:00 2024-03-25T18:47:31.892 INFO:teuthology.orchestra.run.smithi081.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-25T18:47:32.034 INFO:teuthology.orchestra.run.smithi081.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-03-25T18:47:32.093 INFO:teuthology.orchestra.run.smithi081.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-25T18:47:32.301 INFO:teuthology.orchestra.run.smithi081.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-03-25T18:47:32.468 INFO:teuthology.orchestra.run.smithi081.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 4.4 MB 00:00 2024-03-25T18:47:33.893 INFO:teuthology.orchestra.run.smithi081.stdout:(30/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 466 kB/s | 20 MB 00:43 2024-03-25T18:47:34.151 INFO:teuthology.orchestra.run.smithi098.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 298 kB/s | 3.1 MB 00:10 2024-03-25T18:47:42.534 INFO:teuthology.orchestra.run.smithi081.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 318 kB/s | 3.1 MB 00:10 2024-03-25T18:47:47.267 INFO:teuthology.orchestra.run.smithi081.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 231 kB/s | 3.4 MB 00:14 2024-03-25T18:47:47.275 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:47:47.275 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.0 MB/s | 58 MB 00:57 2024-03-25T18:47:47.307 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:47:47.346 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:47:47.346 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:47:47.819 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:47:47.819 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:47:49.290 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:47:49.374 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:47:49.405 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:47:49.430 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:47:49.461 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:47:49.478 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:47:49.671 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:47:49.725 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:47:50.053 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:47:50.120 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:47:50.200 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:47:50.289 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:47:50.317 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:47:50.380 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:47:50.409 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:47:50.453 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:47:50.521 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:47:50.545 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:47:50.592 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:47:50.626 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:47:50.735 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:47:50.777 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:47:50.818 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:47:50.851 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:47:50.896 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:47:50.929 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:47:51.249 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:47:51.284 INFO:teuthology.orchestra.run.smithi081.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:47:51.357 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:47:51.389 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:47:51.434 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:47:51.888 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:47:51.920 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:47:51.992 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:47:52.059 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:47:52.089 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:47:54.231 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:47:54.269 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:47:54.823 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:47:54.864 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:47:54.946 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:47:54.946 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:47:54.946 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:47:54.946 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:47:54.990 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:47:57.124 INFO:teuthology.orchestra.run.smithi098.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 303 kB/s | 20 MB 01:07 2024-03-25T18:47:57.131 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:47:57.131 INFO:teuthology.orchestra.run.smithi098.stdout:Total 875 kB/s | 58 MB 01:07 2024-03-25T18:47:57.161 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:47:57.198 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:47:57.199 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:47:57.669 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:47:57.669 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:47:59.145 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:47:59.229 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:47:59.271 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:47:59.296 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:47:59.319 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:47:59.335 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:47:59.553 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:47:59.607 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:47:59.925 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:47:59.982 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:48:00.056 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:48:00.143 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:48:00.175 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:48:00.237 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:48:00.266 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:48:00.310 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:48:00.377 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:48:00.402 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:48:00.449 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:48:00.483 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:48:00.595 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:48:00.626 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:48:00.668 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:48:00.700 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:48:00.729 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:48:00.761 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:48:01.077 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:48:01.108 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:48:01.181 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:48:01.213 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:48:01.253 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:48:01.703 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:48:01.728 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:48:01.799 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:48:01.866 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:48:01.897 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:48:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:48:04.068 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:48:04.154 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:48:04.154 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /sys 2024-03-25T18:48:04.154 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /proc 2024-03-25T18:48:04.154 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /mnt 2024-03-25T18:48:04.154 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /var/tmp 2024-03-25T18:48:04.154 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /home 2024-03-25T18:48:04.155 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /root 2024-03-25T18:48:04.155 INFO:teuthology.orchestra.run.smithi081.stdout:skipping the directory /tmp 2024-03-25T18:48:04.155 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:48:04.618 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:48:04.655 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:48:04.731 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:48:04.731 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:48:04.731 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:48:04.731 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:48:04.765 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:48:05.070 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:48:05.094 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:48:05.094 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:48:05.094 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:48:05.094 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:48:05.094 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:48:05.095 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:48:05.096 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:48:05.181 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:48:05.181 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:48:06.365 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:48:06.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:48:06.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:48:06.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:48:06.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:48:06.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:48:06.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:48:06.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:48:06.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:48:06.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:48:06.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:48:06.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:48:06.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:48:06.366 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:48:06.367 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:48:06.367 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:48:06.367 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:48:06.367 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:48:06.367 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:48:06.367 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:48:06.367 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:48:06.367 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:48:06.367 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:48:06.367 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:48:06.367 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:48:06.367 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:48:06.367 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:48:06.367 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:48:06.368 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:48:06.368 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:48:06.368 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:48:06.368 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:48:06.368 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:48:06.368 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:48:07.321 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:48:07.321 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:48:07.321 INFO:teuthology.orchestra.run.smithi081.stdout:Upgraded: 2024-03-25T18:48:07.321 INFO:teuthology.orchestra.run.smithi081.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:07.321 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:07.321 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:48:07.321 INFO:teuthology.orchestra.run.smithi081.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:48:07.321 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:07.321 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:07.321 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:07.322 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:07.322 INFO:teuthology.orchestra.run.smithi081.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:48:07.322 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:48:07.322 INFO:teuthology.orchestra.run.smithi081.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:48:07.322 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:07.322 INFO:teuthology.orchestra.run.smithi081.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:48:07.322 INFO:teuthology.orchestra.run.smithi081.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:48:07.322 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:48:07.322 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:07.322 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:48:07.322 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:07.323 INFO:teuthology.orchestra.run.smithi081.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:48:07.323 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:48:07.323 INFO:teuthology.orchestra.run.smithi081.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:48:07.323 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:07.323 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:07.323 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:07.323 INFO:teuthology.orchestra.run.smithi081.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:48:07.323 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:07.323 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:07.323 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:07.323 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:48:07.323 INFO:teuthology.orchestra.run.smithi081.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:48:07.324 INFO:teuthology.orchestra.run.smithi081.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:48:07.324 INFO:teuthology.orchestra.run.smithi081.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:48:07.324 INFO:teuthology.orchestra.run.smithi081.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:48:07.324 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:48:07.324 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:48:07.611 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-test 2024-03-25T18:48:08.169 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:01:21 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:48:08.319 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:48:08.320 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:48:08.320 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T18:48:08.320 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:48:08.320 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:48:08.320 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:48:08.320 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T18:48:08.320 INFO:teuthology.orchestra.run.smithi081.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:48:08.320 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:48:08.321 INFO:teuthology.orchestra.run.smithi081.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:48:08.321 INFO:teuthology.orchestra.run.smithi081.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:48:08.321 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:48:08.321 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:48:08.321 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:48:08.321 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:48:08.321 INFO:teuthology.orchestra.run.smithi081.stdout:Install 6 Packages 2024-03-25T18:48:08.321 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:48:08.322 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 48 M 2024-03-25T18:48:08.322 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 204 M 2024-03-25T18:48:08.322 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:48:08.570 INFO:teuthology.orchestra.run.smithi081.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-25T18:48:08.637 INFO:teuthology.orchestra.run.smithi081.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.8 MB/s | 300 kB 00:00 2024-03-25T18:48:08.662 INFO:teuthology.orchestra.run.smithi081.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-25T18:48:08.796 INFO:teuthology.orchestra.run.smithi081.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-25T18:48:09.571 INFO:teuthology.orchestra.run.smithi081.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 143 kB/s | 163 kB 00:01 2024-03-25T18:48:13.992 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:48:13.992 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-03-25T18:48:13.992 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-03-25T18:48:13.992 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-03-25T18:48:13.992 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-03-25T18:48:13.992 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-03-25T18:48:13.992 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-03-25T18:48:13.992 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-03-25T18:48:13.992 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:48:14.909 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:48:14.933 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:48:14.933 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:48:14.933 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:48:14.933 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:48:14.933 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:48:14.934 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:48:14.935 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:48:15.020 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:48:15.021 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:48:16.215 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:48:16.215 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:48:16.215 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:48:16.215 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:48:16.215 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:48:16.215 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:48:16.215 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:48:16.215 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:48:16.215 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:48:16.215 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:48:16.216 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:48:16.216 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:48:16.216 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:48:16.216 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:48:16.216 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:48:16.216 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:48:16.216 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:48:16.216 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:48:16.216 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:48:16.216 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:48:16.216 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:48:16.216 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:48:16.216 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:48:16.216 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:48:16.217 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:48:16.217 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:48:16.217 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:48:16.217 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:48:16.217 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:48:16.217 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:48:16.218 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:48:16.218 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:48:16.218 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:48:16.218 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:48:17.378 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:48:17.378 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:48:17.378 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-03-25T18:48:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:17.379 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:48:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:48:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:48:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:48:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:48:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:48:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:48:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:48:17.379 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:48:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:48:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:48:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:48:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:48:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:48:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:48:17.380 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:48:17.381 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:48:17.381 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:48:17.381 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:48:17.381 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:48:17.381 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:48:17.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-test 2024-03-25T18:48:18.246 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:01:32 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:48:18.394 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:48:18.395 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:48:18.395 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T18:48:18.395 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:48:18.396 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:48:18.396 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:48:18.396 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T18:48:18.396 INFO:teuthology.orchestra.run.smithi098.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:48:18.396 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:48:18.396 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:48:18.396 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:48:18.396 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:48:18.396 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:48:18.396 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:48:18.396 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:48:18.396 INFO:teuthology.orchestra.run.smithi098.stdout:Install 6 Packages 2024-03-25T18:48:18.397 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:48:18.397 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 48 M 2024-03-25T18:48:18.397 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 204 M 2024-03-25T18:48:18.397 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:48:18.642 INFO:teuthology.orchestra.run.smithi098.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.6 MB/s | 217 kB 00:00 2024-03-25T18:48:18.743 INFO:teuthology.orchestra.run.smithi098.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.6 MB/s | 300 kB 00:00 2024-03-25T18:48:18.777 INFO:teuthology.orchestra.run.smithi098.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T18:48:19.093 INFO:teuthology.orchestra.run.smithi098.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 593 kB/s | 187 kB 00:00 2024-03-25T18:48:19.494 INFO:teuthology.orchestra.run.smithi098.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 166 kB/s | 163 kB 00:00 2024-03-25T18:50:28.857 INFO:teuthology.orchestra.run.smithi098.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 371 kB/s | 47 MB 02:10 2024-03-25T18:50:28.859 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:50:28.859 INFO:teuthology.orchestra.run.smithi098.stdout:Total 378 kB/s | 48 MB 02:10 2024-03-25T18:50:28.869 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:50:28.889 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:50:28.889 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:50:29.264 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:50:29.265 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:50:29.859 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:50:29.947 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:50:29.985 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:50:30.018 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:50:30.059 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:50:30.083 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:50:34.546 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:50:34.616 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:50:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:50:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:50:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:50:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:50:36.983 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:50:36.984 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:50:37.362 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:50:37.362 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:50:37.362 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:50:37.362 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:50:37.362 INFO:teuthology.orchestra.run.smithi098.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:50:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:50:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:50:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:50:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:50:37.363 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:50:37.363 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:50:37.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph 2024-03-25T18:50:38.223 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:52 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:50:38.378 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:50:38.380 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================= 2024-03-25T18:50:38.381 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T18:50:38.381 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================= 2024-03-25T18:50:38.381 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:50:38.381 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:50:38.381 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T18:50:38.381 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:50:38.381 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:50:38.381 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:50:38.382 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:50:38.382 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:50:38.382 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:50:38.382 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:50:38.382 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:50:38.382 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:50:38.382 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:50:38.382 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:50:38.382 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:50:38.383 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:50:38.383 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:50:38.383 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:50:38.383 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:50:38.383 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:50:38.383 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:50:38.383 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:50:38.383 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:50:38.383 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:50:38.383 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:50:38.384 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:50:38.384 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:50:38.384 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:50:38.384 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:50:38.384 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:50:38.384 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:50:38.384 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:50:38.384 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:50:38.384 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:50:38.384 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:50:38.385 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:50:38.385 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:50:38.385 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:50:38.385 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:50:38.385 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:50:38.385 INFO:teuthology.orchestra.run.smithi098.stdout:========================================================================================= 2024-03-25T18:50:38.385 INFO:teuthology.orchestra.run.smithi098.stdout:Install 36 Packages 2024-03-25T18:50:38.385 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:50:38.386 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 30 M 2024-03-25T18:50:38.386 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 107 M 2024-03-25T18:50:38.386 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:50:39.217 INFO:teuthology.orchestra.run.smithi098.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 17 kB/s | 6.6 kB 00:00 2024-03-25T18:50:40.139 INFO:teuthology.orchestra.run.smithi081.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 319 kB/s | 47 MB 02:31 2024-03-25T18:50:40.141 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:50:40.141 INFO:teuthology.orchestra.run.smithi081.stdout:Total 325 kB/s | 48 MB 02:31 2024-03-25T18:50:40.153 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:50:40.172 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:50:40.172 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:50:40.550 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:50:40.551 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:50:41.143 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:50:41.222 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:50:41.269 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:50:41.310 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:50:41.359 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:50:41.383 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:50:42.167 INFO:teuthology.orchestra.run.smithi098.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 456 kB/s | 1.5 MB 00:03 2024-03-25T18:50:45.701 INFO:teuthology.orchestra.run.smithi098.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 316 kB/s | 2.1 MB 00:06 2024-03-25T18:50:45.849 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:50:45.934 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:50:46.409 INFO:teuthology.orchestra.run.smithi098.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 349 kB/s | 247 kB 00:00 2024-03-25T18:50:46.568 INFO:teuthology.orchestra.run.smithi098.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-25T18:50:46.610 INFO:teuthology.orchestra.run.smithi098.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T18:50:46.643 INFO:teuthology.orchestra.run.smithi098.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 945 kB/s | 32 kB 00:00 2024-03-25T18:50:46.669 INFO:teuthology.orchestra.run.smithi098.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-25T18:50:46.710 INFO:teuthology.orchestra.run.smithi098.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 931 kB/s | 39 kB 00:00 2024-03-25T18:50:46.769 INFO:teuthology.orchestra.run.smithi098.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-25T18:50:46.861 INFO:teuthology.orchestra.run.smithi098.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 13 MB/s | 1.1 MB 00:00 2024-03-25T18:50:46.903 INFO:teuthology.orchestra.run.smithi098.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T18:50:46.945 INFO:teuthology.orchestra.run.smithi098.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T18:50:46.987 INFO:teuthology.orchestra.run.smithi098.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T18:50:47.154 INFO:teuthology.orchestra.run.smithi098.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 749 kB/s | 125 kB 00:00 2024-03-25T18:50:47.254 INFO:teuthology.orchestra.run.smithi098.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-03-25T18:50:47.629 INFO:teuthology.orchestra.run.smithi098.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 403 kB/s | 151 kB 00:00 2024-03-25T18:50:47.713 INFO:teuthology.orchestra.run.smithi098.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 521 kB/s | 43 kB 00:00 2024-03-25T18:50:47.830 INFO:teuthology.orchestra.run.smithi098.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-03-25T18:50:47.956 INFO:teuthology.orchestra.run.smithi098.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-03-25T18:50:48.022 INFO:teuthology.orchestra.run.smithi098.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 161 kB/s | 11 kB 00:00 2024-03-25T18:50:48.098 INFO:teuthology.orchestra.run.smithi098.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-03-25T18:50:48.165 INFO:teuthology.orchestra.run.smithi098.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 347 kB/s | 23 kB 00:00 2024-03-25T18:50:48.232 INFO:teuthology.orchestra.run.smithi098.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 292 kB/s | 19 kB 00:00 2024-03-25T18:50:48.299 INFO:teuthology.orchestra.run.smithi098.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 295 kB/s | 20 kB 00:00 2024-03-25T18:50:48.366 INFO:teuthology.orchestra.run.smithi098.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 695 kB/s | 46 kB 00:00 2024-03-25T18:50:48.441 INFO:teuthology.orchestra.run.smithi098.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-25T18:50:48.474 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:50:48.474 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:50:48.474 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:50:48.475 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:50:48.475 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:50:48.475 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:50:48.525 INFO:teuthology.orchestra.run.smithi098.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-25T18:50:48.600 INFO:teuthology.orchestra.run.smithi098.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 219 kB/s | 16 kB 00:00 2024-03-25T18:50:48.675 INFO:teuthology.orchestra.run.smithi098.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-25T18:50:48.742 INFO:teuthology.orchestra.run.smithi098.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 536 kB/s | 36 kB 00:00 2024-03-25T18:50:48.826 INFO:teuthology.orchestra.run.smithi098.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-25T18:50:48.838 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:50:48.838 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:50:48.838 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:50:48.839 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:50:48.839 INFO:teuthology.orchestra.run.smithi081.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:50:48.839 INFO:teuthology.orchestra.run.smithi081.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:50:48.839 INFO:teuthology.orchestra.run.smithi081.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:50:48.839 INFO:teuthology.orchestra.run.smithi081.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:50:48.839 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:50:48.839 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:50:48.839 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:50:48.901 INFO:teuthology.orchestra.run.smithi098.stdout:(33/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 495 kB/s | 4.7 MB 00:09 2024-03-25T18:50:48.927 INFO:teuthology.orchestra.run.smithi098.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-03-25T18:50:48.977 INFO:teuthology.orchestra.run.smithi098.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 268 kB/s | 20 kB 00:00 2024-03-25T18:50:49.134 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph 2024-03-25T18:50:49.698 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:04:02 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:50:49.858 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:50:49.861 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================= 2024-03-25T18:50:49.861 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T18:50:49.861 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================= 2024-03-25T18:50:49.861 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:50:49.861 INFO:teuthology.orchestra.run.smithi081.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:50:49.861 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T18:50:49.862 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:50:49.862 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:50:49.862 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:50:49.862 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:50:49.862 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:50:49.862 INFO:teuthology.orchestra.run.smithi081.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:50:49.862 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:50:49.862 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:50:49.862 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:50:49.862 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:50:49.863 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:50:49.863 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:50:49.863 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:50:49.863 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:50:49.863 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:50:49.863 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:50:49.863 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:50:49.863 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:50:49.863 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:50:49.863 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:50:49.863 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:50:49.863 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:50:49.864 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:50:49.864 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:50:49.864 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:50:49.864 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:50:49.864 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:50:49.864 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:50:49.864 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:50:49.864 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:50:49.864 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:50:49.864 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:50:49.864 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:50:49.864 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:50:49.864 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:50:49.865 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:50:49.865 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:50:49.865 INFO:teuthology.orchestra.run.smithi081.stdout:========================================================================================= 2024-03-25T18:50:49.865 INFO:teuthology.orchestra.run.smithi081.stdout:Install 36 Packages 2024-03-25T18:50:49.865 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:50:49.865 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 30 M 2024-03-25T18:50:49.866 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 107 M 2024-03-25T18:50:49.866 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:50:50.792 INFO:teuthology.orchestra.run.smithi081.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 14 kB/s | 6.6 kB 00:00 2024-03-25T18:50:54.109 INFO:teuthology.orchestra.run.smithi081.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 572 kB/s | 2.1 MB 00:03 2024-03-25T18:50:57.201 INFO:teuthology.orchestra.run.smithi081.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 221 kB/s | 1.5 MB 00:06 2024-03-25T18:50:57.976 INFO:teuthology.orchestra.run.smithi081.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 319 kB/s | 247 kB 00:00 2024-03-25T18:50:58.134 INFO:teuthology.orchestra.run.smithi081.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-25T18:50:58.193 INFO:teuthology.orchestra.run.smithi081.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-25T18:50:58.235 INFO:teuthology.orchestra.run.smithi081.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 757 kB/s | 32 kB 00:00 2024-03-25T18:50:58.293 INFO:teuthology.orchestra.run.smithi081.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 811 kB/s | 47 kB 00:00 2024-03-25T18:50:58.335 INFO:teuthology.orchestra.run.smithi081.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 930 kB/s | 39 kB 00:00 2024-03-25T18:50:58.386 INFO:teuthology.orchestra.run.smithi081.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T18:50:58.453 INFO:teuthology.orchestra.run.smithi081.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T18:50:58.503 INFO:teuthology.orchestra.run.smithi081.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-25T18:50:58.536 INFO:teuthology.orchestra.run.smithi081.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T18:50:58.587 INFO:teuthology.orchestra.run.smithi081.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-25T18:50:59.020 INFO:teuthology.orchestra.run.smithi081.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 288 kB/s | 125 kB 00:00 2024-03-25T18:50:59.129 INFO:teuthology.orchestra.run.smithi081.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 206 kB/s | 22 kB 00:00 2024-03-25T18:50:59.755 INFO:teuthology.orchestra.run.smithi081.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 242 kB/s | 151 kB 00:00 2024-03-25T18:50:59.863 INFO:teuthology.orchestra.run.smithi081.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 401 kB/s | 43 kB 00:00 2024-03-25T18:51:00.130 INFO:teuthology.orchestra.run.smithi081.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 645 kB/s | 172 kB 00:00 2024-03-25T18:51:00.647 INFO:teuthology.orchestra.run.smithi081.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 694 kB/s | 358 kB 00:00 2024-03-25T18:51:00.755 INFO:teuthology.orchestra.run.smithi081.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 99 kB/s | 11 kB 00:00 2024-03-25T18:51:00.864 INFO:teuthology.orchestra.run.smithi081.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-03-25T18:51:00.964 INFO:teuthology.orchestra.run.smithi081.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 232 kB/s | 23 kB 00:00 2024-03-25T18:51:01.065 INFO:teuthology.orchestra.run.smithi081.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-03-25T18:51:01.165 INFO:teuthology.orchestra.run.smithi081.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 197 kB/s | 20 kB 00:00 2024-03-25T18:51:01.273 INFO:teuthology.orchestra.run.smithi081.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 428 kB/s | 46 kB 00:00 2024-03-25T18:51:01.382 INFO:teuthology.orchestra.run.smithi081.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 728 kB/s | 79 kB 00:00 2024-03-25T18:51:01.732 INFO:teuthology.orchestra.run.smithi081.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 778 kB/s | 272 kB 00:00 2024-03-25T18:51:01.849 INFO:teuthology.orchestra.run.smithi081.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 141 kB/s | 16 kB 00:00 2024-03-25T18:51:02.066 INFO:teuthology.orchestra.run.smithi081.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 415 kB/s | 90 kB 00:00 2024-03-25T18:51:02.192 INFO:teuthology.orchestra.run.smithi081.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 285 kB/s | 36 kB 00:00 2024-03-25T18:51:02.692 INFO:teuthology.orchestra.run.smithi081.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 459 kB/s | 230 kB 00:00 2024-03-25T18:51:03.634 INFO:teuthology.orchestra.run.smithi081.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 454 kB/s | 427 kB 00:00 2024-03-25T18:51:03.792 INFO:teuthology.orchestra.run.smithi081.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 126 kB/s | 20 kB 00:00 2024-03-25T18:51:03.926 INFO:teuthology.orchestra.run.smithi081.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 365 kB/s | 4.7 MB 00:13 2024-03-25T18:51:18.275 INFO:teuthology.orchestra.run.smithi098.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 476 kB/s | 17 MB 00:36 2024-03-25T18:51:18.279 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:51:18.279 INFO:teuthology.orchestra.run.smithi098.stdout:Total 761 kB/s | 30 MB 00:39 2024-03-25T18:51:18.302 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:51:18.342 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:51:18.342 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:51:18.741 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:51:18.741 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:51:20.047 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:51:20.110 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:51:20.147 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:51:20.179 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:51:20.249 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:51:20.286 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:51:20.319 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:51:20.411 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:51:20.496 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:51:20.538 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:51:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:51:20.664 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:51:20.702 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:51:20.751 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:51:20.815 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:51:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:51:21.040 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:51:21.070 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:51:21.105 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:51:21.135 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:51:21.211 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:51:21.245 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:51:21.279 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:51:21.319 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:51:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:51:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:51:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:51:21.503 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:51:21.574 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:51:21.728 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:51:21.854 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:51:22.013 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:51:22.041 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:51:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:51:22.066 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:51:22.066 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:51:22.067 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:51:22.067 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:51:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:23.535 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:51:23.562 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:51:23.562 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:51:23.562 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:51:23.562 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:51:23.562 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:51:23.563 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:24.010 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:51:24.034 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:51:24.034 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:51:24.034 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:51:24.035 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:51:24.035 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:51:24.035 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:24.253 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:51:24.276 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:51:24.276 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:51:24.276 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:51:24.276 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:51:24.276 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:51:24.276 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:24.344 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:51:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:51:26.076 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:51:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:51:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:51:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:51:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:51:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:51:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:51:26.077 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:51:26.078 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:51:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:51:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:51:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:51:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:51:26.081 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:51:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:51:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:51:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:51:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:51:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:51:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:51:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:51:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:51:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:51:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:51:26.082 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:51:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:51:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:51:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:51:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:51:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:51:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:51:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:51:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:51:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:51:26.083 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:51:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:51:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:26.941 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:51:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:51:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:51:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:51:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:51:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:51:26.941 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:51:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:51:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:51:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:51:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:51:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:51:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:51:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:51:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:51:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:51:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:51:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:51:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:51:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:51:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:51:26.942 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:51:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:51:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:51:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:51:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:51:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:51:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:51:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:51:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:51:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:51:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:51:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:51:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:51:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:51:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:51:26.943 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:51:26.944 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:26.944 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:51:27.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-base 2024-03-25T18:51:27.827 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:04:41 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:51:27.924 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:51:27.976 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:51:27.977 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T18:51:27.978 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:51:28.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install cephadm 2024-03-25T18:51:28.593 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:04:42 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:51:28.741 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:51:28.742 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:51:28.743 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T18:51:28.743 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:51:28.743 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:51:28.743 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:51:28.743 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:28.743 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:51:28.743 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:51:28.743 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T18:51:28.744 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:28.744 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 747 k 2024-03-25T18:51:28.744 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 754 k 2024-03-25T18:51:28.744 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:51:30.032 INFO:teuthology.orchestra.run.smithi081.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 478 kB/s | 17 MB 00:35 2024-03-25T18:51:30.036 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:51:30.036 INFO:teuthology.orchestra.run.smithi081.stdout:Total 756 kB/s | 30 MB 00:40 2024-03-25T18:51:30.058 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:51:30.099 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:51:30.099 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:51:30.501 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:51:30.502 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:51:31.281 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 294 kB/s | 747 kB 00:02 2024-03-25T18:51:31.287 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:51:31.288 INFO:teuthology.orchestra.run.smithi098.stdout:Total 294 kB/s | 747 kB 00:02 2024-03-25T18:51:31.288 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:51:31.289 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:51:31.289 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:51:31.295 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:51:31.295 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:51:31.371 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:51:31.834 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:51:31.870 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:51:31.928 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:51:31.934 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:51:31.981 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:51:32.020 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:51:32.090 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:51:32.136 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:51:32.169 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:51:32.260 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:51:32.289 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:51:32.347 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:51:32.388 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:51:32.460 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:51:32.513 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:51:32.552 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:51:32.577 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:51:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:32.578 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:51:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:51:32.578 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:32.578 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:51:32.601 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:51:32.656 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:51:32.785 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:51:32.850 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:51:32.898 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:51:32.936 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:51:32.971 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:51:33.001 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:51:33.069 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:51:33.103 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:51:33.136 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:51:33.176 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:51:33.212 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:51:33.255 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:51:33.344 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:51:33.347 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:04:47 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:51:33.385 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:51:33.458 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:51:33.498 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:51:33.499 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:51:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T18:51:33.499 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:51:33.499 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:51:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:51:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:33.499 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:51:33.499 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:51:33.499 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T18:51:33.499 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:33.500 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 145 k 2024-03-25T18:51:33.500 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 436 k 2024-03-25T18:51:33.500 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:51:33.594 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:51:33.720 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:51:33.879 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:51:33.907 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:51:33.932 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:51:33.932 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:51:33.932 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:51:33.933 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:51:33.933 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:51:33.933 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:34.263 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 190 kB/s | 145 kB 00:00 2024-03-25T18:51:34.264 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:51:34.265 INFO:teuthology.orchestra.run.smithi098.stdout:Total 190 kB/s | 145 kB 00:00 2024-03-25T18:51:34.265 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:51:34.273 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:51:34.273 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:51:34.325 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:51:34.326 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:51:34.468 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:51:34.531 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:51:34.555 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:51:34.555 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:51:34.555 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:51:34.556 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:35.367 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:51:35.393 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:51:35.393 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:51:35.393 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:51:35.393 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:51:35.393 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:51:35.393 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:51:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:35.479 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:51:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:51:35.479 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:35.479 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:51:35.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr 2024-03-25T18:51:35.834 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:51:35.857 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:51:35.857 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:51:35.857 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:51:35.857 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:51:35.857 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:51:35.857 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:36.060 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:51:36.082 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:51:36.083 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:51:36.083 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:51:36.083 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:51:36.083 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:51:36.083 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:36.148 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:51:36.308 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:04:50 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:51:36.406 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:51:36.458 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:51:36.459 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T18:51:36.459 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:51:36.520 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:51:37.077 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:04:51 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:51:37.226 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:51:37.227 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-03-25T18:51:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T18:51:37.228 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-03-25T18:51:37.228 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:51:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:51:37.228 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T18:51:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:51:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:51:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:51:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:51:37.228 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:51:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:51:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:51:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:51:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:51:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:51:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:51:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:37.229 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:51:37.229 INFO:teuthology.orchestra.run.smithi098.stdout:====================================================================================== 2024-03-25T18:51:37.229 INFO:teuthology.orchestra.run.smithi098.stdout:Install 12 Packages 2024-03-25T18:51:37.229 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:37.230 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 6.9 M 2024-03-25T18:51:37.230 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 26 M 2024-03-25T18:51:37.230 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:51:37.753 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:51:37.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:51:37.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:51:37.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:51:37.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:51:37.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:51:37.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:51:37.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:51:37.754 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:51:37.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:51:37.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:51:37.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:51:37.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:51:37.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:51:37.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:51:37.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:51:37.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:51:37.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:51:37.755 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:51:37.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:51:37.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:51:37.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:51:37.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:51:37.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:51:37.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:51:37.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:51:37.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:51:37.756 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:51:37.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:51:37.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:51:37.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:51:37.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:51:37.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:51:37.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:51:37.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:51:37.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:51:38.218 INFO:teuthology.orchestra.run.smithi098.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 60 kB/s | 31 kB 00:00 2024-03-25T18:51:38.368 INFO:teuthology.orchestra.run.smithi098.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-03-25T18:51:38.410 INFO:teuthology.orchestra.run.smithi098.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 5.6 MB/s | 238 kB 00:00 2024-03-25T18:51:38.435 INFO:teuthology.orchestra.run.smithi098.stdout:(4/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 21 kB/s | 15 kB 00:00 2024-03-25T18:51:38.727 INFO:teuthology.orchestra.run.smithi098.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.7 MB/s | 863 kB 00:00 2024-03-25T18:51:38.819 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:51:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:38.820 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:51:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:51:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:51:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:51:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:51:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:51:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:51:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:51:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:51:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:51:38.820 INFO:teuthology.orchestra.run.smithi081.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:51:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:51:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:51:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:51:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:51:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:51:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:51:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:51:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:51:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:51:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:51:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:51:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:51:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:51:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:51:38.821 INFO:teuthology.orchestra.run.smithi081.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:51:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:51:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:51:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:51:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:51:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:51:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:51:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:51:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:51:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:51:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:51:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:51:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:38.822 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:51:39.111 INFO:teuthology.orchestra.run.smithi098.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 51 kB/s | 19 kB 00:00 2024-03-25T18:51:39.181 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-base 2024-03-25T18:51:39.186 INFO:teuthology.orchestra.run.smithi098.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 730 kB/s | 548 kB 00:00 2024-03-25T18:51:39.470 INFO:teuthology.orchestra.run.smithi098.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 510 kB/s | 144 kB 00:00 2024-03-25T18:51:39.636 INFO:teuthology.orchestra.run.smithi098.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 185 kB/s | 31 kB 00:00 2024-03-25T18:51:39.746 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:04:52 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:51:39.762 INFO:teuthology.orchestra.run.smithi098.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.1 MB/s | 2.0 MB 00:00 2024-03-25T18:51:39.804 INFO:teuthology.orchestra.run.smithi098.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.1 MB/s | 188 kB 00:00 2024-03-25T18:51:39.842 INFO:teuthology.orchestra.run.smithi081.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:51:39.893 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:51:39.894 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T18:51:39.894 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:51:39.958 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install cephadm 2024-03-25T18:51:40.520 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:04:53 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:51:40.669 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:51:40.670 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:51:40.670 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T18:51:40.670 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:51:40.671 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:51:40.671 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:51:40.671 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:40.671 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:51:40.671 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:51:40.671 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T18:51:40.671 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:40.671 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 747 k 2024-03-25T18:51:40.671 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 754 k 2024-03-25T18:51:40.671 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:51:42.985 INFO:teuthology.orchestra.run.smithi081.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 323 kB/s | 747 kB 00:02 2024-03-25T18:51:42.986 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:51:42.986 INFO:teuthology.orchestra.run.smithi081.stdout:Total 323 kB/s | 747 kB 00:02 2024-03-25T18:51:42.986 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:51:42.992 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:51:42.992 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:51:42.998 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:51:42.999 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:51:43.077 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:51:43.522 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:51:43.607 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:51:44.008 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:51:44.305 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:51:44.305 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:44.305 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:51:44.306 INFO:teuthology.orchestra.run.smithi081.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:51:44.306 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:44.306 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:51:44.504 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:51:44.653 INFO:teuthology.orchestra.run.smithi098.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 262 kB/s | 1.8 MB 00:06 2024-03-25T18:51:44.658 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:51:44.658 INFO:teuthology.orchestra.run.smithi098.stdout:Total 945 kB/s | 6.9 MB 00:07 2024-03-25T18:51:44.679 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:51:44.697 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:51:44.697 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:51:44.877 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:51:44.877 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:51:45.070 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:04:58 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:51:45.226 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:51:45.226 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:51:45.226 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T18:51:45.226 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:51:45.226 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:51:45.227 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:51:45.227 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:45.227 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:51:45.227 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:51:45.227 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T18:51:45.227 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:45.227 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 145 k 2024-03-25T18:51:45.227 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 436 k 2024-03-25T18:51:45.228 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:51:45.383 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:51:45.470 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:51:45.514 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:51:45.560 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:51:45.592 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:51:45.684 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:51:45.817 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:51:45.910 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:51:45.954 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:51:45.983 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:51:46.014 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:51:46.074 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 172 kB/s | 145 kB 00:00 2024-03-25T18:51:46.075 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:51:46.075 INFO:teuthology.orchestra.run.smithi081.stdout:Total 171 kB/s | 145 kB 00:00 2024-03-25T18:51:46.076 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:51:46.084 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:51:46.084 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:51:46.138 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:51:46.139 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:51:46.247 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:51:46.281 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:51:46.282 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:51:46.342 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:51:46.366 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:51:46.366 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:51:46.366 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:51:46.366 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:47.102 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:51:47.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:51:47.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:51:47.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:51:47.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:51:47.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:51:47.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:51:47.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:51:47.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:51:47.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:51:47.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:51:47.103 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:51:47.363 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:51:47.363 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:47.363 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:51:47.363 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:51:47.363 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:47.364 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:51:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:51:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:47.444 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:51:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:51:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:51:47.444 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:51:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:51:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:51:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:51:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:51:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:51:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:51:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:51:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:51:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:51:47.445 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:47.445 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:51:47.590 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr 2024-03-25T18:51:47.714 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:51:48.160 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:05:01 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:51:48.258 INFO:teuthology.orchestra.run.smithi081.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:51:48.268 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:05:02 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:51:48.310 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:51:48.311 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T18:51:48.311 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:51:48.390 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:51:48.419 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-25T18:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T18:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-25T18:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:51:48.420 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T18:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================ 2024-03-25T18:51:48.421 INFO:teuthology.orchestra.run.smithi098.stdout:Install 12 Packages 2024-03-25T18:51:48.422 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:51:48.422 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 37 M 2024-03-25T18:51:48.422 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 209 M 2024-03-25T18:51:48.422 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:51:48.862 INFO:teuthology.orchestra.run.smithi098.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 237 kB/s | 32 kB 00:00 2024-03-25T18:51:48.920 INFO:teuthology.orchestra.run.smithi098.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 499 kB/s | 17 kB 00:00 2024-03-25T18:51:48.946 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:05:01 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:51:48.953 INFO:teuthology.orchestra.run.smithi098.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T18:51:49.095 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:51:49.096 INFO:teuthology.orchestra.run.smithi081.stdout:====================================================================================== 2024-03-25T18:51:49.096 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T18:51:49.096 INFO:teuthology.orchestra.run.smithi081.stdout:====================================================================================== 2024-03-25T18:51:49.096 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:51:49.096 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:51:49.096 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T18:51:49.096 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:51:49.097 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:51:49.097 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:51:49.097 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:51:49.097 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:51:49.097 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:51:49.097 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:51:49.097 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:51:49.097 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:51:49.097 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:51:49.097 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:51:49.097 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:49.097 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:51:49.097 INFO:teuthology.orchestra.run.smithi081.stdout:====================================================================================== 2024-03-25T18:51:49.097 INFO:teuthology.orchestra.run.smithi081.stdout:Install 12 Packages 2024-03-25T18:51:49.098 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:49.098 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 6.9 M 2024-03-25T18:51:49.098 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 26 M 2024-03-25T18:51:49.098 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:51:49.112 INFO:teuthology.orchestra.run.smithi098.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-25T18:51:49.254 INFO:teuthology.orchestra.run.smithi098.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-25T18:51:49.296 INFO:teuthology.orchestra.run.smithi098.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-03-25T18:51:49.346 INFO:teuthology.orchestra.run.smithi098.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.8 MB/s | 3.0 MB 00:00 2024-03-25T18:51:49.521 INFO:teuthology.orchestra.run.smithi098.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-25T18:51:49.672 INFO:teuthology.orchestra.run.smithi098.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-25T18:51:49.678 INFO:teuthology.orchestra.run.smithi081.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 77 kB/s | 31 kB 00:00 2024-03-25T18:51:49.697 INFO:teuthology.orchestra.run.smithi098.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-25T18:51:49.779 INFO:teuthology.orchestra.run.smithi081.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 30 kB/s | 15 kB 00:00 2024-03-25T18:51:49.814 INFO:teuthology.orchestra.run.smithi098.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-25T18:51:49.821 INFO:teuthology.orchestra.run.smithi081.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-03-25T18:51:49.846 INFO:teuthology.orchestra.run.smithi081.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 3.5 MB/s | 238 kB 00:00 2024-03-25T18:51:50.104 INFO:teuthology.orchestra.run.smithi081.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.1 MB/s | 548 kB 00:00 2024-03-25T18:51:50.138 INFO:teuthology.orchestra.run.smithi081.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.7 MB/s | 863 kB 00:00 2024-03-25T18:51:50.247 INFO:teuthology.orchestra.run.smithi081.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 137 kB/s | 19 kB 00:00 2024-03-25T18:51:50.389 INFO:teuthology.orchestra.run.smithi081.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.0 MB/s | 144 kB 00:00 2024-03-25T18:51:50.497 INFO:teuthology.orchestra.run.smithi081.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 284 kB/s | 31 kB 00:00 2024-03-25T18:51:50.623 INFO:teuthology.orchestra.run.smithi081.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-25T18:51:50.823 INFO:teuthology.orchestra.run.smithi081.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.1 MB/s | 2.0 MB 00:00 2024-03-25T18:51:55.465 INFO:teuthology.orchestra.run.smithi081.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 294 kB/s | 1.8 MB 00:06 2024-03-25T18:51:55.468 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:51:55.468 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.1 MB/s | 6.9 MB 00:06 2024-03-25T18:51:55.487 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:51:55.507 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:51:55.507 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:51:55.696 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:51:55.697 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:51:56.223 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:51:56.298 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:51:56.333 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:51:56.380 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:51:56.410 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:51:56.510 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:51:56.635 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:51:56.721 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:51:56.773 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:51:56.801 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:51:56.833 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:51:57.075 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:51:57.109 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:51:57.865 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:51:57.865 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:51:57.866 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:51:57.866 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:51:57.866 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:51:57.866 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:51:57.866 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:51:57.866 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:51:57.866 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:51:57.866 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:51:57.866 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:51:57.866 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:51:58.228 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:51:58.228 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:58.229 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:51:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:51:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:51:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:51:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:51:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:51:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:51:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:51:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:51:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:51:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:51:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:51:58.229 INFO:teuthology.orchestra.run.smithi081.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:51:58.230 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:58.230 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:51:58.491 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:51:59.046 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:51:59.199 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:51:59.200 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================================ 2024-03-25T18:51:59.201 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T18:51:59.201 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================================ 2024-03-25T18:51:59.201 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:51:59.201 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:51:59.201 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T18:51:59.201 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:51:59.201 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:51:59.201 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:51:59.201 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:51:59.201 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:51:59.202 INFO:teuthology.orchestra.run.smithi081.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:51:59.202 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:51:59.202 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:51:59.202 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:51:59.202 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:51:59.202 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:51:59.202 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:59.202 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:51:59.202 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================================ 2024-03-25T18:51:59.202 INFO:teuthology.orchestra.run.smithi081.stdout:Install 12 Packages 2024-03-25T18:51:59.202 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:51:59.203 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 37 M 2024-03-25T18:51:59.203 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 209 M 2024-03-25T18:51:59.203 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:51:59.713 INFO:teuthology.orchestra.run.smithi081.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 272 kB/s | 32 kB 00:00 2024-03-25T18:51:59.771 INFO:teuthology.orchestra.run.smithi081.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 494 kB/s | 17 kB 00:00 2024-03-25T18:51:59.805 INFO:teuthology.orchestra.run.smithi081.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T18:51:59.980 INFO:teuthology.orchestra.run.smithi081.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-25T18:52:00.105 INFO:teuthology.orchestra.run.smithi081.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.9 MB/s | 3.0 MB 00:00 2024-03-25T18:52:00.172 INFO:teuthology.orchestra.run.smithi081.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2024-03-25T18:52:00.197 INFO:teuthology.orchestra.run.smithi081.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T18:52:00.356 INFO:teuthology.orchestra.run.smithi081.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.0 MB/s | 803 kB 00:00 2024-03-25T18:52:00.548 INFO:teuthology.orchestra.run.smithi081.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-25T18:52:00.573 INFO:teuthology.orchestra.run.smithi081.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 911 kB/s | 195 kB 00:00 2024-03-25T18:52:00.765 INFO:teuthology.orchestra.run.smithi081.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-25T18:52:10.679 INFO:teuthology.orchestra.run.smithi098.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 345 kB/s | 7.4 MB 00:21 2024-03-25T18:52:10.681 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:10.681 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.7 MB/s | 37 MB 00:22 2024-03-25T18:52:10.725 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:52:10.751 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:52:10.751 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:52:11.100 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:52:11.100 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:52:11.805 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:52:11.876 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:52:11.908 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:52:12.004 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:52:12.036 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:52:12.205 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:52:12.235 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:52:12.316 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:52:12.971 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:52:13.073 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:52:14.305 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:52:15.490 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:52:15.727 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:52:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:52:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:52:18.070 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:52:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:52:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:52:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:52:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:52:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:52:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:52:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:52:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:52:18.071 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:52:18.548 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:52:18.548 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:18.548 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:52:18.549 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:52:18.549 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:52:18.549 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:52:18.549 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:52:18.549 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:52:18.549 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:52:18.549 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:52:18.549 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:52:18.549 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:52:18.549 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:52:18.549 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:52:18.549 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:52:18.549 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:18.549 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:52:18.900 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:52:19.466 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:05:33 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:52:19.615 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:52:19.616 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T18:52:19.616 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T18:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T18:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T18:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:52:19.618 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:52:19.618 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:19.618 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:52:19.618 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T18:52:19.618 INFO:teuthology.orchestra.run.smithi098.stdout:Install 13 Packages 2024-03-25T18:52:19.618 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:19.618 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 2.0 M 2024-03-25T18:52:19.619 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 26 M 2024-03-25T18:52:19.619 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:52:20.073 INFO:teuthology.orchestra.run.smithi098.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 236 kB/s | 24 kB 00:00 2024-03-25T18:52:20.123 INFO:teuthology.orchestra.run.smithi098.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 118 kB/s | 18 kB 00:00 2024-03-25T18:52:20.165 INFO:teuthology.orchestra.run.smithi098.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-25T18:52:20.191 INFO:teuthology.orchestra.run.smithi098.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T18:52:20.216 INFO:teuthology.orchestra.run.smithi098.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:52:20.241 INFO:teuthology.orchestra.run.smithi098.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 871 kB/s | 43 kB 00:00 2024-03-25T18:52:20.475 INFO:teuthology.orchestra.run.smithi098.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2024-03-25T18:52:20.500 INFO:teuthology.orchestra.run.smithi098.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 113 kB/s | 32 kB 00:00 2024-03-25T18:52:20.609 INFO:teuthology.orchestra.run.smithi098.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 82 kB/s | 52 kB 00:00 2024-03-25T18:52:20.826 INFO:teuthology.orchestra.run.smithi098.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 578 kB/s | 203 kB 00:00 2024-03-25T18:52:20.951 INFO:teuthology.orchestra.run.smithi098.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 173 kB/s | 59 kB 00:00 2024-03-25T18:52:20.976 INFO:teuthology.orchestra.run.smithi098.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 597 kB/s | 90 kB 00:00 2024-03-25T18:52:21.018 INFO:teuthology.orchestra.run.smithi098.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.0 MB/s | 1.0 MB 00:00 2024-03-25T18:52:21.020 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:21.021 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-25T18:52:21.021 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:52:21.036 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:52:21.036 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:52:21.133 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:52:21.134 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:52:21.643 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:52:21.804 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:52:21.860 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:52:21.923 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:52:21.953 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:52:21.995 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:52:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:52:22.205 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:52:22.254 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:52:22.969 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:52:23.011 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:52:23.044 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:52:23.093 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:52:23.115 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:52:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:52:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:52:23.919 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:52:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:52:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:52:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:52:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:52:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:52:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:52:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:52:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:52:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:52:23.920 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:52:24.279 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:52:24.279 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:24.279 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:52:24.279 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:52:24.279 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:52:24.279 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:52:24.279 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:52:24.280 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:52:24.280 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:52:24.280 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:52:24.280 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:52:24.280 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:52:24.280 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:52:24.280 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:52:24.280 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:52:24.280 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:52:24.280 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:24.280 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:52:24.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:52:25.104 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:05:39 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:52:25.255 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:52:25.256 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T18:52:25.256 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T18:52:25.256 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T18:52:25.256 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:52:25.256 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:52:25.256 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T18:52:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:52:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:52:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:52:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:52:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:52:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:25.257 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:52:25.257 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T18:52:25.257 INFO:teuthology.orchestra.run.smithi098.stdout:Install 6 Packages 2024-03-25T18:52:25.257 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:25.258 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 6.8 M 2024-03-25T18:52:25.258 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 33 M 2024-03-25T18:52:25.258 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:52:25.646 INFO:teuthology.orchestra.run.smithi098.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-25T18:52:25.671 INFO:teuthology.orchestra.run.smithi098.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 886 kB/s | 248 kB 00:00 2024-03-25T18:52:25.988 INFO:teuthology.orchestra.run.smithi098.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 182 kB/s | 58 kB 00:00 2024-03-25T18:52:26.097 INFO:teuthology.orchestra.run.smithi098.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 471 kB/s | 51 kB 00:00 2024-03-25T18:52:26.130 INFO:teuthology.orchestra.run.smithi098.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-25T18:52:26.813 INFO:teuthology.orchestra.run.smithi081.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 278 kB/s | 7.4 MB 00:27 2024-03-25T18:52:26.816 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:26.816 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.3 MB/s | 37 MB 00:27 2024-03-25T18:52:26.858 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:52:26.884 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:52:26.884 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:52:27.249 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:52:27.250 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:52:27.946 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:52:28.018 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:52:28.050 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:52:28.142 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:52:28.203 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:52:28.373 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:52:28.402 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:52:28.479 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:52:28.513 INFO:teuthology.orchestra.run.smithi098.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 44 kB/s | 138 kB 00:03 2024-03-25T18:52:28.516 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:28.517 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.1 MB/s | 6.8 MB 00:03 2024-03-25T18:52:28.532 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:52:28.543 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:52:28.544 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:52:28.635 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:52:28.636 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:52:28.939 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:52:29.105 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:52:29.139 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:52:29.157 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:52:29.247 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:52:29.682 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:52:29.759 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:52:29.822 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:52:29.853 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:52:30.480 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:52:30.584 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:52:30.584 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:52:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:52:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:52:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:52:30.585 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:52:30.888 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:52:30.888 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:30.888 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:52:30.888 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:52:30.888 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:52:30.889 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:52:30.889 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:52:30.889 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:52:30.889 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:52:30.889 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:30.889 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:52:31.127 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-fuse 2024-03-25T18:52:31.496 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:52:31.527 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:52:31.692 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:05:45 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:52:31.843 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:52:31.843 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:31.843 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T18:52:31.843 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:31.843 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:52:31.843 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:52:31.844 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T18:52:31.844 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:52:31.844 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:31.844 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:52:31.844 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:31.844 INFO:teuthology.orchestra.run.smithi098.stdout:Install 2 Packages 2024-03-25T18:52:31.844 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:31.844 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 964 k 2024-03-25T18:52:31.845 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 2.8 M 2024-03-25T18:52:31.845 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:52:31.957 INFO:teuthology.orchestra.run.smithi098.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 699 kB/s | 79 kB 00:00 2024-03-25T18:52:34.215 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:52:34.216 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:52:34.216 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:52:34.216 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:52:34.216 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:52:34.216 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:52:34.216 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:52:34.216 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:52:34.216 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:52:34.216 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:52:34.217 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:52:34.217 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:52:34.636 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:52:34.636 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:52:34.637 INFO:teuthology.orchestra.run.smithi081.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:34.638 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:52:35.007 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:52:35.576 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:05:48 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:52:35.725 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:52:35.727 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T18:52:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T18:52:35.727 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T18:52:35.727 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:52:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:52:35.727 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T18:52:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:52:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:52:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:52:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:52:35.727 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:52:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:52:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:52:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:52:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:52:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:52:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:52:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:52:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:35.728 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:52:35.728 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T18:52:35.728 INFO:teuthology.orchestra.run.smithi081.stdout:Install 13 Packages 2024-03-25T18:52:35.728 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:35.729 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 2.0 M 2024-03-25T18:52:35.729 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 26 M 2024-03-25T18:52:35.729 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:52:35.987 INFO:teuthology.orchestra.run.smithi081.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 232 kB/s | 24 kB 00:00 2024-03-25T18:52:36.038 INFO:teuthology.orchestra.run.smithi081.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 116 kB/s | 18 kB 00:00 2024-03-25T18:52:36.080 INFO:teuthology.orchestra.run.smithi081.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-25T18:52:36.105 INFO:teuthology.orchestra.run.smithi081.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T18:52:36.130 INFO:teuthology.orchestra.run.smithi081.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:52:36.155 INFO:teuthology.orchestra.run.smithi081.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 872 kB/s | 43 kB 00:00 2024-03-25T18:52:36.389 INFO:teuthology.orchestra.run.smithi081.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 124 kB/s | 32 kB 00:00 2024-03-25T18:52:36.414 INFO:teuthology.orchestra.run.smithi081.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-03-25T18:52:36.565 INFO:teuthology.orchestra.run.smithi081.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.1 MB/s | 203 kB 00:00 2024-03-25T18:52:36.590 INFO:teuthology.orchestra.run.smithi081.stdout:(10/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 74 kB/s | 52 kB 00:00 2024-03-25T18:52:36.690 INFO:teuthology.orchestra.run.smithi081.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 3.7 MB/s | 1.0 MB 00:00 2024-03-25T18:52:36.715 INFO:teuthology.orchestra.run.smithi081.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 393 kB/s | 59 kB 00:00 2024-03-25T18:52:36.740 INFO:teuthology.orchestra.run.smithi081.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 597 kB/s | 90 kB 00:00 2024-03-25T18:52:36.744 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:36.745 INFO:teuthology.orchestra.run.smithi081.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-25T18:52:36.745 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:52:36.758 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:52:36.758 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:52:36.799 INFO:teuthology.orchestra.run.smithi098.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 179 kB/s | 885 kB 00:04 2024-03-25T18:52:36.801 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:36.801 INFO:teuthology.orchestra.run.smithi098.stdout:Total 194 kB/s | 964 kB 00:04 2024-03-25T18:52:36.801 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:52:36.825 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:52:36.825 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:52:36.848 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:52:36.850 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:52:36.902 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:52:36.903 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:52:37.086 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:52:37.216 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:52:37.290 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:52:37.362 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:52:37.529 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:52:37.583 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:52:37.639 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:52:37.676 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:52:37.726 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:52:37.820 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:52:37.920 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:52:37.977 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:52:37.990 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:52:37.990 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:52:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:52:38.288 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:38.288 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:52:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:52:38.289 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:38.289 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:52:38.587 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-volume 2024-03-25T18:52:38.704 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:52:38.758 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:52:38.791 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:52:38.842 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:52:38.863 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:52:39.165 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:05:53 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:52:39.315 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:52:39.316 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:39.316 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-03-25T18:52:39.317 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:39.317 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:52:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:52:39.317 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T18:52:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:52:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:39.317 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:52:39.317 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:39.317 INFO:teuthology.orchestra.run.smithi098.stdout:Install 2 Packages 2024-03-25T18:52:39.317 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:39.318 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 332 k 2024-03-25T18:52:39.318 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 1.5 M 2024-03-25T18:52:39.318 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:52:39.429 INFO:teuthology.orchestra.run.smithi098.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 619 kB/s | 69 kB 00:00 2024-03-25T18:52:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:52:39.633 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:52:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:52:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:52:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:52:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:52:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:52:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:52:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:52:39.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:52:39.635 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:52:39.635 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:52:39.635 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:52:39.991 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:52:39.991 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:39.991 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:52:39.991 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:52:39.991 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:52:39.992 INFO:teuthology.orchestra.run.smithi081.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:52:39.992 INFO:teuthology.orchestra.run.smithi081.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:52:39.992 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:52:39.992 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:52:39.992 INFO:teuthology.orchestra.run.smithi081.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:52:39.992 INFO:teuthology.orchestra.run.smithi081.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:52:39.992 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:52:39.992 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:52:39.992 INFO:teuthology.orchestra.run.smithi081.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:52:39.993 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:52:39.993 INFO:teuthology.orchestra.run.smithi081.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:52:39.993 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:39.993 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:52:40.276 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:52:40.746 INFO:teuthology.orchestra.run.smithi098.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 185 kB/s | 264 kB 00:01 2024-03-25T18:52:40.747 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:40.748 INFO:teuthology.orchestra.run.smithi098.stdout:Total 232 kB/s | 332 kB 00:01 2024-03-25T18:52:40.748 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:52:40.758 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:52:40.758 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:52:40.804 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:52:40.805 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:52:40.837 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:05:53 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:52:40.955 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:52:40.986 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:52:40.987 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T18:52:40.987 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T18:52:40.987 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T18:52:40.988 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:52:40.988 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:52:40.988 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T18:52:40.988 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:52:40.988 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:52:40.988 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:52:40.988 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:52:40.988 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:52:40.988 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:40.988 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:52:40.988 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T18:52:40.989 INFO:teuthology.orchestra.run.smithi081.stdout:Install 6 Packages 2024-03-25T18:52:40.989 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:40.989 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 6.8 M 2024-03-25T18:52:40.989 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 33 M 2024-03-25T18:52:40.989 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:52:41.154 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:52:41.182 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:52:41.197 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:52:41.197 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:52:41.197 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:52:41.197 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:41.628 INFO:teuthology.orchestra.run.smithi081.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-25T18:52:41.770 INFO:teuthology.orchestra.run.smithi081.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-25T18:52:41.864 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:52:41.903 INFO:teuthology.orchestra.run.smithi081.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 434 kB/s | 58 kB 00:00 2024-03-25T18:52:41.953 INFO:teuthology.orchestra.run.smithi081.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2024-03-25T18:52:42.012 INFO:teuthology.orchestra.run.smithi081.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 468 kB/s | 51 kB 00:00 2024-03-25T18:52:42.172 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:52:42.172 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:42.172 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:52:42.172 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:52:42.172 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:52:42.172 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:42.172 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:52:42.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados-devel 2024-03-25T18:52:42.549 INFO:teuthology.orchestra.run.smithi081.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 126 kB/s | 138 kB 00:01 2024-03-25T18:52:42.551 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:42.551 INFO:teuthology.orchestra.run.smithi081.stdout:Total 4.3 MB/s | 6.8 MB 00:01 2024-03-25T18:52:42.564 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:52:42.575 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:52:42.575 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:52:42.663 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:52:42.663 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:52:42.968 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:52:43.088 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:05:57 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:52:43.139 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:52:43.188 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:52:43.236 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:52:43.237 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:43.237 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T18:52:43.237 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:43.237 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:52:43.237 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:52:43.237 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:43.237 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:52:43.237 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:43.237 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T18:52:43.237 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:43.238 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 127 k 2024-03-25T18:52:43.238 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 456 k 2024-03-25T18:52:43.238 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:52:43.703 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:52:43.782 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:52:43.853 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:52:43.876 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:52:44.077 INFO:teuthology.orchestra.run.smithi098.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 151 kB/s | 127 kB 00:00 2024-03-25T18:52:44.077 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:44.078 INFO:teuthology.orchestra.run.smithi098.stdout:Total 151 kB/s | 127 kB 00:00 2024-03-25T18:52:44.078 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:52:44.084 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:52:44.084 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:52:44.143 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:52:44.143 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:52:44.291 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:52:44.401 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:52:44.634 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:52:44.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:52:44.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:52:44.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:52:44.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:52:44.634 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:52:44.854 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:52:44.931 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:52:44.931 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:44.931 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:52:44.932 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:52:44.932 INFO:teuthology.orchestra.run.smithi081.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:52:44.932 INFO:teuthology.orchestra.run.smithi081.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:52:44.932 INFO:teuthology.orchestra.run.smithi081.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:52:44.932 INFO:teuthology.orchestra.run.smithi081.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:52:44.932 INFO:teuthology.orchestra.run.smithi081.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:52:44.932 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:44.932 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:52:45.144 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:52:45.144 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:45.144 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:52:45.144 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:52:45.144 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:45.144 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:52:45.201 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-fuse 2024-03-25T18:52:45.380 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs2 2024-03-25T18:52:45.764 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:05:58 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:52:45.913 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:52:45.914 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:52:45.914 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T18:52:45.914 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:52:45.914 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:52:45.914 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:52:45.914 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T18:52:45.914 INFO:teuthology.orchestra.run.smithi081.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:52:45.914 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:45.914 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:52:45.914 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:52:45.914 INFO:teuthology.orchestra.run.smithi081.stdout:Install 2 Packages 2024-03-25T18:52:45.914 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:45.915 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 964 k 2024-03-25T18:52:45.915 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 2.8 M 2024-03-25T18:52:45.915 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:52:45.951 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:05:59 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:52:46.022 INFO:teuthology.orchestra.run.smithi081.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 734 kB/s | 79 kB 00:00 2024-03-25T18:52:46.050 INFO:teuthology.orchestra.run.smithi098.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:52:46.107 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:52:46.108 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T18:52:46.108 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:52:46.167 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs-devel 2024-03-25T18:52:46.762 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:06:00 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:52:46.918 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:52:46.919 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T18:52:46.919 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:46.919 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:52:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:52:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:46.919 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:52:46.919 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:46.919 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T18:52:46.919 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:46.919 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 31 k 2024-03-25T18:52:46.920 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 140 k 2024-03-25T18:52:46.920 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:52:47.482 INFO:teuthology.orchestra.run.smithi098.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 55 kB/s | 31 kB 00:00 2024-03-25T18:52:47.482 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:47.483 INFO:teuthology.orchestra.run.smithi098.stdout:Total 55 kB/s | 31 kB 00:00 2024-03-25T18:52:47.483 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:52:47.486 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:52:47.486 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:52:47.511 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:52:47.511 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:52:47.634 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:52:47.746 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:52:48.139 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:52:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:52:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:48.393 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:52:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:52:48.393 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:48.393 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:52:48.554 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados2 2024-03-25T18:52:48.563 INFO:teuthology.orchestra.run.smithi081.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 334 kB/s | 885 kB 00:02 2024-03-25T18:52:48.565 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:48.565 INFO:teuthology.orchestra.run.smithi081.stdout:Total 363 kB/s | 964 kB 00:02 2024-03-25T18:52:48.566 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:52:48.589 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:52:48.590 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:52:48.667 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:52:48.668 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:52:48.848 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:52:49.015 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:52:49.080 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:52:49.119 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:06:03 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:52:49.217 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:52:49.268 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:52:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T18:52:49.269 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:52:49.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librbd1 2024-03-25T18:52:49.781 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:52:49.781 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:52:49.883 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:06:03 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:52:49.979 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:52:50.029 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:52:50.031 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T18:52:50.031 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:52:50.070 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:52:50.070 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:50.071 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:52:50.071 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:52:50.071 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:50.071 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:52:50.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rados 2024-03-25T18:52:50.385 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-volume 2024-03-25T18:52:50.643 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:06:04 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:52:50.740 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:52:50.792 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:52:50.793 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T18:52:50.793 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:52:50.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rgw 2024-03-25T18:52:50.954 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:06:03 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:52:51.106 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:52:51.106 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:52:51.107 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2024-03-25T18:52:51.107 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:52:51.107 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:52:51.107 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:52:51.107 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T18:52:51.107 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:52:51.107 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:51.107 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:52:51.107 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:52:51.107 INFO:teuthology.orchestra.run.smithi081.stdout:Install 2 Packages 2024-03-25T18:52:51.107 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:51.108 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 332 k 2024-03-25T18:52:51.108 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 1.5 M 2024-03-25T18:52:51.108 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:52:51.218 INFO:teuthology.orchestra.run.smithi081.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 626 kB/s | 69 kB 00:00 2024-03-25T18:52:51.409 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:06:05 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:52:51.506 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:52:51.558 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:52:51.560 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T18:52:51.560 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:52:51.612 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-cephfs 2024-03-25T18:52:52.043 INFO:teuthology.orchestra.run.smithi081.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 282 kB/s | 264 kB 00:00 2024-03-25T18:52:52.044 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:52.045 INFO:teuthology.orchestra.run.smithi081.stdout:Total 354 kB/s | 332 kB 00:00 2024-03-25T18:52:52.045 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:52:52.054 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:52:52.054 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:52:52.093 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:52:52.094 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:52:52.175 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:06:06 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:52:52.244 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:52:52.272 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:52:52.324 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:52:52.325 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T18:52:52.326 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:52:52.368 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rbd 2024-03-25T18:52:52.419 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:52:52.454 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:52:52.470 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:52:52.470 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:52:52.470 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:52:52.470 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:52.925 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:06:06 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:52:53.021 INFO:teuthology.orchestra.run.smithi098.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:52:53.073 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:52:53.074 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T18:52:53.074 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:52:53.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-fuse 2024-03-25T18:52:53.189 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:52:53.540 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:52:53.540 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:53.540 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:52:53.540 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:52:53.540 INFO:teuthology.orchestra.run.smithi081.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:52:53.540 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:53.540 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:52:53.698 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:06:07 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:52:53.848 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:52:53.848 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-03-25T18:52:53.848 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:53.848 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:52:53.848 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:52:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:53.849 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:52:53.849 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:53.849 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T18:52:53.849 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:53.849 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 85 k 2024-03-25T18:52:53.849 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 231 k 2024-03-25T18:52:53.849 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:52:53.882 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librados-devel 2024-03-25T18:52:54.460 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:06:07 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:52:54.560 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 120 kB/s | 85 kB 00:00 2024-03-25T18:52:54.560 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:54.560 INFO:teuthology.orchestra.run.smithi098.stdout:Total 120 kB/s | 85 kB 00:00 2024-03-25T18:52:54.561 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:52:54.568 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:52:54.568 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:52:54.610 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:52:54.610 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:52:54.610 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T18:52:54.610 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:52:54.610 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:52:54.611 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:52:54.611 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:54.611 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:52:54.611 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:52:54.611 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T18:52:54.611 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:54.611 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 127 k 2024-03-25T18:52:54.612 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 456 k 2024-03-25T18:52:54.612 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:52:54.625 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:52:54.625 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:52:54.766 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:52:54.876 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:52:55.274 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:52:55.283 INFO:teuthology.orchestra.run.smithi081.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 189 kB/s | 127 kB 00:00 2024-03-25T18:52:55.283 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:55.284 INFO:teuthology.orchestra.run.smithi081.stdout:Total 188 kB/s | 127 kB 00:00 2024-03-25T18:52:55.284 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:52:55.289 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:52:55.290 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:52:55.346 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:52:55.347 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:52:55.489 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:52:55.565 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:52:55.566 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:55.566 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:52:55.566 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:52:55.566 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:55.566 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:52:55.598 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:52:55.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-mirror 2024-03-25T18:52:56.073 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:52:56.335 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:06:10 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:52:56.356 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:52:56.356 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:56.357 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:52:56.357 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:52:56.357 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:56.357 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:52:56.488 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:52:56.488 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:56.488 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T18:52:56.488 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:56.489 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:52:56.489 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:52:56.489 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:56.489 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:52:56.489 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:52:56.489 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T18:52:56.489 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:52:56.490 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 3.1 M 2024-03-25T18:52:56.490 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 13 M 2024-03-25T18:52:56.490 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:52:56.631 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install libcephfs2 2024-03-25T18:52:57.204 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:06:10 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:52:57.302 INFO:teuthology.orchestra.run.smithi081.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:52:57.357 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:52:57.358 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T18:52:57.358 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:52:57.427 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install libcephfs-devel 2024-03-25T18:52:57.991 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:06:10 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:52:58.139 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:52:58.139 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:52:58.140 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T18:52:58.140 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:52:58.140 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:52:58.140 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:52:58.140 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:58.140 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:52:58.140 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:52:58.140 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T18:52:58.140 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:58.140 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 31 k 2024-03-25T18:52:58.140 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 140 k 2024-03-25T18:52:58.140 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:52:58.586 INFO:teuthology.orchestra.run.smithi081.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 70 kB/s | 31 kB 00:00 2024-03-25T18:52:58.587 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:52:58.587 INFO:teuthology.orchestra.run.smithi081.stdout:Total 70 kB/s | 31 kB 00:00 2024-03-25T18:52:58.587 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:52:58.590 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:52:58.590 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:52:58.628 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:52:58.628 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:52:58.769 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:52:58.860 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:52:59.271 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:52:59.530 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:52:59.530 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:59.530 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:52:59.530 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:52:59.531 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:52:59.531 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:52:59.687 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librados2 2024-03-25T18:53:00.246 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:06:13 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:53:00.342 INFO:teuthology.orchestra.run.smithi081.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:53:00.394 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:53:00.395 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T18:53:00.395 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:53:00.444 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librbd1 2024-03-25T18:53:01.004 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:06:14 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:53:01.100 INFO:teuthology.orchestra.run.smithi081.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:53:01.151 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:53:01.152 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T18:53:01.152 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:53:01.197 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rados 2024-03-25T18:53:01.759 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:06:14 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:53:01.854 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:53:01.908 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:53:01.909 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T18:53:01.909 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:53:01.960 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rgw 2024-03-25T18:53:02.555 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:06:15 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:53:02.657 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:53:02.711 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:53:02.712 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T18:53:02.712 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:53:02.757 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-cephfs 2024-03-25T18:53:03.327 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:06:16 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:53:03.422 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:53:03.473 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:53:03.474 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T18:53:03.475 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:53:03.517 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rbd 2024-03-25T18:53:04.076 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:06:17 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:53:04.171 INFO:teuthology.orchestra.run.smithi081.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:53:04.222 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:53:04.223 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T18:53:04.223 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:53:04.267 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-fuse 2024-03-25T18:53:04.830 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:06:17 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:53:04.978 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:53:04.979 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:53:04.979 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-03-25T18:53:04.979 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:53:04.979 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:53:04.979 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:53:04.980 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:53:04.980 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:53:04.980 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:53:04.980 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T18:53:04.980 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:53:04.980 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 85 k 2024-03-25T18:53:04.980 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 231 k 2024-03-25T18:53:04.980 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:53:06.424 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 59 kB/s | 85 kB 00:01 2024-03-25T18:53:06.424 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:53:06.424 INFO:teuthology.orchestra.run.smithi081.stdout:Total 59 kB/s | 85 kB 00:01 2024-03-25T18:53:06.424 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:53:06.432 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:53:06.433 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:53:06.494 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:53:06.494 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:53:06.633 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:53:06.738 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:53:06.771 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 308 kB/s | 3.1 MB 00:10 2024-03-25T18:53:06.772 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:53:06.772 INFO:teuthology.orchestra.run.smithi098.stdout:Total 307 kB/s | 3.1 MB 00:10 2024-03-25T18:53:06.772 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:53:06.783 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:53:06.783 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:53:06.859 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:53:06.860 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:53:07.149 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:53:07.284 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:53:07.329 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:53:07.354 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:53:07.354 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:53:07.354 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:53:07.354 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:53:07.354 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:53:07.354 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:53:07.444 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:53:07.444 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:53:07.444 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:53:07.444 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:53:07.444 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:53:07.444 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:53:07.699 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-mirror 2024-03-25T18:53:08.268 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:06:21 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:53:08.421 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:53:08.422 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:53:08.422 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T18:53:08.422 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:53:08.423 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:53:08.423 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:53:08.423 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:53:08.423 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:53:08.423 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:53:08.423 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T18:53:08.423 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:53:08.423 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 3.1 M 2024-03-25T18:53:08.423 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 13 M 2024-03-25T18:53:08.423 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:53:08.482 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:53:08.482 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:53:08.482 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:53:08.483 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:53:08.483 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:53:08.483 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:53:08.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-nbd 2024-03-25T18:53:09.325 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:06:23 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:53:09.473 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:53:09.474 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:53:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-03-25T18:53:09.474 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:53:09.474 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:53:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:53:09.474 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:53:09.474 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:53:09.474 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:53:09.474 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T18:53:09.475 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:53:09.475 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 171 k 2024-03-25T18:53:09.476 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 494 k 2024-03-25T18:53:09.476 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:53:10.775 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 132 kB/s | 171 kB 00:01 2024-03-25T18:53:10.775 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:53:10.775 INFO:teuthology.orchestra.run.smithi098.stdout:Total 132 kB/s | 171 kB 00:01 2024-03-25T18:53:10.776 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:53:10.785 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:53:10.785 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:53:10.845 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:53:10.845 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:53:10.996 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:53:11.106 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:53:11.567 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:53:11.866 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:53:11.867 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:53:11.867 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:53:11.867 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:53:11.867 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:53:11.867 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:53:12.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:53:12.724 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:06:26 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:53:12.876 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:53:12.876 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:53:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T18:53:12.877 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:53:12.877 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:53:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:53:12.877 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-25T18:53:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:53:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:53:12.877 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:53:12.877 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:53:12.877 INFO:teuthology.orchestra.run.smithi098.stdout:Install 2 Packages 2024-03-25T18:53:12.877 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:53:12.878 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 91 M 2024-03-25T18:53:12.878 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 498 M 2024-03-25T18:53:12.878 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:53:13.701 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 599 kB/s | 3.1 MB 00:05 2024-03-25T18:53:13.702 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:53:13.702 INFO:teuthology.orchestra.run.smithi081.stdout:Total 599 kB/s | 3.1 MB 00:05 2024-03-25T18:53:13.702 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:53:13.713 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:53:13.713 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:53:13.789 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:53:13.789 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:53:14.219 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:53:14.294 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:53:14.318 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:53:14.318 INFO:teuthology.orchestra.run.smithi081.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:53:14.318 INFO:teuthology.orchestra.run.smithi081.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:53:14.318 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:53:14.319 INFO:teuthology.orchestra.run.smithi081.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:53:14.319 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:53:15.459 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:53:15.459 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:53:15.459 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:53:15.459 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:53:15.459 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:53:15.459 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:53:15.704 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-nbd 2024-03-25T18:53:16.277 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:06:29 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:53:16.429 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:53:16.430 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:53:16.430 INFO:teuthology.orchestra.run.smithi081.stdout: Package Architecture Version Repository Size 2024-03-25T18:53:16.430 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:53:16.430 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:53:16.430 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:53:16.430 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:53:16.430 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:53:16.430 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:53:16.430 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T18:53:16.430 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:53:16.430 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 171 k 2024-03-25T18:53:16.430 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 494 k 2024-03-25T18:53:16.430 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:53:17.505 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 160 kB/s | 171 kB 00:01 2024-03-25T18:53:17.505 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:53:17.506 INFO:teuthology.orchestra.run.smithi081.stdout:Total 159 kB/s | 171 kB 00:01 2024-03-25T18:53:17.506 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:53:17.515 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:53:17.515 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:53:17.575 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:53:17.576 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:53:17.728 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:53:17.836 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:53:18.271 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:53:18.567 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:53:18.567 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:53:18.567 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:53:18.568 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:53:18.568 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:53:18.568 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:53:18.814 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:53:19.385 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:06:32 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:53:19.535 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:53:19.536 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:53:19.536 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T18:53:19.536 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:53:19.536 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:53:19.536 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:53:19.537 INFO:teuthology.orchestra.run.smithi081.stdout:Installing dependencies: 2024-03-25T18:53:19.537 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:53:19.537 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:53:19.537 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:53:19.537 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:53:19.537 INFO:teuthology.orchestra.run.smithi081.stdout:Install 2 Packages 2024-03-25T18:53:19.537 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:53:19.537 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 91 M 2024-03-25T18:53:19.537 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 498 M 2024-03-25T18:53:19.537 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T18:54:13.739 INFO:teuthology.orchestra.run.smithi098.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 378 kB/s | 22 MB 01:00 2024-03-25T18:54:32.859 INFO:teuthology.orchestra.run.smithi081.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 313 kB/s | 22 MB 01:13 2024-03-25T18:56:09.693 INFO:teuthology.orchestra.run.smithi098.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 395 kB/s | 68 MB 02:56 2024-03-25T18:56:09.694 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:56:09.694 INFO:teuthology.orchestra.run.smithi098.stdout:Total 525 kB/s | 91 MB 02:56 2024-03-25T18:56:09.694 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T18:56:09.698 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T18:56:09.698 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T18:56:10.284 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T18:56:10.285 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T18:56:13.449 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T18:56:20.555 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:56:20.627 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:56:24.132 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:56:24.133 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:56:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:56:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:56:24.397 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T18:56:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:56:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:56:24.397 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:56:24.397 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T18:56:24.438 INFO:teuthology.orchestra.run.smithi081.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 377 kB/s | 68 MB 03:04 2024-03-25T18:56:24.440 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:56:24.440 INFO:teuthology.orchestra.run.smithi081.stdout:Total 502 kB/s | 91 MB 03:04 2024-03-25T18:56:24.440 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T18:56:24.444 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T18:56:24.444 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T18:56:24.634 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:56:25.027 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T18:56:25.028 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T18:56:25.193 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:09:39 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T18:56:25.342 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T18:56:25.343 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:56:25.343 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T18:56:25.343 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:56:25.343 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T18:56:25.343 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:56:25.343 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:56:25.343 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T18:56:25.343 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T18:56:25.344 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T18:56:25.344 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T18:56:25.344 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 416 M 2024-03-25T18:56:25.344 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 1.7 G 2024-03-25T18:56:25.344 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T18:56:28.176 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T18:56:35.266 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:56:35.422 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:56:38.823 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:56:38.824 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:56:39.090 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:56:39.090 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:56:39.090 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T18:56:39.090 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:56:39.090 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:56:39.090 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:56:39.090 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T18:56:39.312 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:56:39.870 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:09:52 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T18:56:40.019 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T18:56:40.020 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:56:40.020 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T18:56:40.020 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:56:40.020 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T18:56:40.021 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:56:40.021 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:56:40.021 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T18:56:40.021 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T18:56:40.021 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T18:56:40.021 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T18:56:40.021 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 416 M 2024-03-25T18:56:40.021 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 1.7 G 2024-03-25T18:56:40.021 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:09:15.712 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 564 kB/s | 416 MB 12:35 2024-03-25T19:09:15.713 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:09:15.713 INFO:teuthology.orchestra.run.smithi081.stdout:Total 564 kB/s | 416 MB 12:35 2024-03-25T19:09:15.713 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:09:15.717 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:09:15.717 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:09:18.184 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:09:18.184 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:10:00.492 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:10:00.607 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:10:08.697 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:10:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:10:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:10:08.960 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:10:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:10:08.960 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:10:08.960 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:10:09.244 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T19:10:09.806 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:23:22 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:10:09.953 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:10:09.954 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T19:10:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:10:09.954 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T19:10:09.954 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:10:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T19:10:09.954 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:10:09.954 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:10:09.954 INFO:teuthology.orchestra.run.smithi081.stdout:======================================================================================== 2024-03-25T19:10:09.955 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:10:09.955 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:10:09.955 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 1.4 M 2024-03-25T19:10:09.955 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 6.4 M 2024-03-25T19:10:09.955 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:10:16.261 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 232 kB/s | 1.4 MB 00:06 2024-03-25T19:10:16.261 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:10:16.261 INFO:teuthology.orchestra.run.smithi081.stdout:Total 232 kB/s | 1.4 MB 00:06 2024-03-25T19:10:16.262 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:10:16.266 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:10:16.266 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:10:16.310 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:10:16.311 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:10:16.561 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:10:16.701 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T19:10:17.118 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T19:10:17.389 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T19:10:17.389 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:10:17.389 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:10:17.389 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:10:17.389 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:10:17.389 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:10:17.600 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T19:10:18.156 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:23:31 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:10:18.304 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:10:18.305 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:10:18.305 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:10:18.305 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:10:18.305 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:10:18.305 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T19:10:18.305 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:10:18.305 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:10:18.305 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:10:18.305 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:10:18.305 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:10:18.305 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 176 M 2024-03-25T19:10:18.306 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 753 M 2024-03-25T19:10:18.306 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:13:29.153 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 416 kB/s | 416 MB 17:03 2024-03-25T19:13:29.153 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:13:29.153 INFO:teuthology.orchestra.run.smithi098.stdout:Total 416 kB/s | 416 MB 17:03 2024-03-25T19:13:29.153 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:13:29.156 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:13:29.156 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:13:31.623 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:13:31.623 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:14:13.735 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:14:13.839 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:14:18.949 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:14:19.230 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:14:19.231 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:14:19.231 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:14:19.231 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:14:19.231 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:14:19.231 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:14:19.505 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T19:14:20.079 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:27:34 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:14:20.236 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:14:20.236 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T19:14:20.237 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:14:20.237 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T19:14:20.237 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:14:20.237 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T19:14:20.237 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:14:20.237 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:14:20.237 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================== 2024-03-25T19:14:20.237 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:14:20.237 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:14:20.237 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 1.4 M 2024-03-25T19:14:20.237 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 6.4 M 2024-03-25T19:14:20.237 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:14:23.168 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 499 kB/s | 1.4 MB 00:02 2024-03-25T19:14:23.169 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:14:23.169 INFO:teuthology.orchestra.run.smithi098.stdout:Total 498 kB/s | 1.4 MB 00:02 2024-03-25T19:14:23.169 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:14:23.173 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:14:23.173 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:14:23.216 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:14:23.217 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:14:23.483 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:14:23.582 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T19:14:23.972 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T19:14:24.238 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T19:14:24.239 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:14:24.239 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:14:24.239 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:14:24.239 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:14:24.239 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:14:24.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T19:14:24.979 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:27:38 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:14:25.127 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:14:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:14:25.128 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:14:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:14:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:14:25.128 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T19:14:25.128 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:14:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:14:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:14:25.128 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:14:25.129 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:14:25.129 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 176 M 2024-03-25T19:14:25.129 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 753 M 2024-03-25T19:14:25.129 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:17:28.726 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 418 kB/s | 176 MB 07:10 2024-03-25T19:17:28.727 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:17:28.727 INFO:teuthology.orchestra.run.smithi081.stdout:Total 418 kB/s | 176 MB 07:10 2024-03-25T19:17:28.727 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:17:28.730 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:17:28.730 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:17:29.793 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:17:29.794 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:17:47.788 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:17:47.956 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:17:52.514 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:17:52.792 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:17:52.792 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:17:52.792 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:17:52.792 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:17:52.792 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:17:52.792 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:17:53.013 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-test-debuginfo 2024-03-25T19:17:53.574 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:31:06 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:17:53.722 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:17:53.722 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:17:53.722 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:17:53.722 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:17:53.722 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:17:53.722 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T19:17:53.722 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:17:53.723 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:17:53.723 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:17:53.723 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:17:53.723 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:17:53.723 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 672 M 2024-03-25T19:17:53.723 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 3.1 G 2024-03-25T19:17:53.723 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:22:02.247 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 394 kB/s | 176 MB 07:37 2024-03-25T19:22:02.247 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:22:02.247 INFO:teuthology.orchestra.run.smithi098.stdout:Total 394 kB/s | 176 MB 07:37 2024-03-25T19:22:02.247 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:22:02.250 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:22:02.251 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:22:03.311 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:22:03.312 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:22:21.617 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:22:21.793 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:22:26.443 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:22:26.699 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:22:26.699 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:22:26.699 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:22:26.699 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:22:26.700 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:22:26.700 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:22:26.926 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-test-debuginfo 2024-03-25T19:22:27.485 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:35:41 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:22:27.633 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:22:27.633 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:22:27.633 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:22:27.634 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:22:27.634 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:22:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T19:22:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:22:27.634 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:22:27.634 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:22:27.634 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:22:27.634 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:22:27.635 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 672 M 2024-03-25T19:22:27.635 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 3.1 G 2024-03-25T19:22:27.635 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:32:39.848 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 777 kB/s | 672 MB 14:46 2024-03-25T19:32:39.848 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:39.848 INFO:teuthology.orchestra.run.smithi081.stdout:Total 777 kB/s | 672 MB 14:46 2024-03-25T19:32:39.848 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:32:39.853 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:32:39.853 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:32:43.843 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:32:43.843 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:33:54.704 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:33:54.807 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:34:04.657 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:34:04.922 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:34:04.922 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:34:04.922 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:34:04.923 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:34:04.923 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:34:04.923 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:34:05.214 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:34:05.779 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:47:18 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:34:05.875 INFO:teuthology.orchestra.run.smithi081.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:34:05.927 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:34:05.928 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do. 2024-03-25T19:34:05.928 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:34:05.998 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:34:06.559 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:47:19 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:34:06.706 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:34:06.707 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:34:06.707 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:34:06.707 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:34:06.707 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:34:06.707 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:34:06.707 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:34:06.707 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:34:06.707 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:34:06.707 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:34:06.707 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:34:06.708 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 20 M 2024-03-25T19:34:06.708 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 103 M 2024-03-25T19:34:06.708 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:34:17.927 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 969 kB/s | 672 MB 11:50 2024-03-25T19:34:17.928 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:34:17.928 INFO:teuthology.orchestra.run.smithi098.stdout:Total 969 kB/s | 672 MB 11:50 2024-03-25T19:34:17.928 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:34:17.933 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:34:17.933 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:34:21.889 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:34:21.889 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:34:30.067 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 864 kB/s | 20 MB 00:23 2024-03-25T19:34:30.068 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:34:30.069 INFO:teuthology.orchestra.run.smithi081.stdout:Total 863 kB/s | 20 MB 00:23 2024-03-25T19:34:30.069 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:34:30.073 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:34:30.073 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:34:30.224 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:34:30.224 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:34:32.456 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:34:32.567 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:34:33.566 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:34:33.826 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:34:33.826 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:34:33.826 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:34:33.826 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:34:33.826 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:34:33.826 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:34:34.030 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:34:34.592 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:47:47 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:34:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:34:34.740 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:34:34.740 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:34:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:34:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:34:34.741 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:34:34.741 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:34:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:34:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:34:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:34:34.741 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:34:34.741 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 48 M 2024-03-25T19:34:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 173 M 2024-03-25T19:34:34.742 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:34:47.227 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 3.8 MB/s | 48 MB 00:12 2024-03-25T19:34:47.228 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:34:47.228 INFO:teuthology.orchestra.run.smithi081.stdout:Total 3.8 MB/s | 48 MB 00:12 2024-03-25T19:34:47.229 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:34:47.233 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:34:47.233 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:34:47.546 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:34:47.546 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:34:52.287 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:34:52.383 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:34:53.765 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:34:54.024 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:34:54.024 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:34:54.024 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:34:54.024 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:34:54.024 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:34:54.024 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:34:54.241 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:34:54.800 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:48:07 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:34:54.949 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:34:54.950 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:34:54.950 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:34:54.950 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:34:54.950 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:34:54.950 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:34:54.950 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:34:54.950 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:34:54.950 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:34:54.951 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:34:54.951 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:34:54.951 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 99 M 2024-03-25T19:34:54.951 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 449 M 2024-03-25T19:34:54.951 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:35:32.608 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:35:33.166 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:35:42.192 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:35:42.441 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:35:42.441 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:35:42.441 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:35:42.441 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:35:42.441 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:35:42.441 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:35:42.727 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:35:43.292 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:48:57 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:35:43.389 INFO:teuthology.orchestra.run.smithi098.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:35:43.440 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:35:43.441 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-03-25T19:35:43.441 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:35:43.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:35:44.081 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:48:58 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:35:44.230 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:35:44.231 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:35:44.231 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:35:44.231 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:35:44.231 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:35:44.231 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:35:44.232 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:35:44.232 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:35:44.232 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:35:44.232 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:35:44.232 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:35:44.232 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 20 M 2024-03-25T19:35:44.232 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 103 M 2024-03-25T19:35:44.232 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:35:59.982 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 20 MB 00:15 2024-03-25T19:35:59.983 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:35:59.984 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.3 MB/s | 20 MB 00:15 2024-03-25T19:35:59.984 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:35:59.988 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:35:59.988 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:36:00.139 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:36:00.139 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:36:02.369 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:36:02.465 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:36:03.459 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:36:03.715 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:36:03.715 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:36:03.715 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:36:03.715 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:36:03.715 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:36:03.715 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:36:03.905 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:36:04.465 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:49:18 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:36:04.613 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:36:04.614 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:36:04.614 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:36:04.614 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:36:04.614 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:36:04.614 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:36:04.614 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:36:04.614 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:36:04.614 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:36:04.614 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:36:04.614 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:36:04.615 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 48 M 2024-03-25T19:36:04.615 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 173 M 2024-03-25T19:36:04.615 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:36:07.239 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.4 MB/s | 99 MB 01:12 2024-03-25T19:36:07.239 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:36:07.240 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.4 MB/s | 99 MB 01:12 2024-03-25T19:36:07.240 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:36:07.243 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:36:07.244 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:36:07.852 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:36:07.852 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:36:18.132 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:36:18.237 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:36:21.248 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:36:21.504 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:36:21.505 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:36:21.505 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:36:21.505 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:36:21.505 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:36:21.505 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:36:21.706 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:36:22.269 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:49:35 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:36:22.419 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:36:22.419 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:36:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:36:22.420 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:36:22.420 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:36:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:36:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:36:22.420 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:36:22.420 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:36:22.420 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:36:22.420 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:36:22.420 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 287 M 2024-03-25T19:36:22.420 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 1.2 G 2024-03-25T19:36:22.420 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:36:37.348 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.5 MB/s | 48 MB 00:32 2024-03-25T19:36:37.349 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:36:37.349 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.5 MB/s | 48 MB 00:32 2024-03-25T19:36:37.350 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:36:37.353 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:36:37.353 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:36:37.666 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:36:37.667 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:36:42.412 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:36:42.512 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:36:43.855 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:36:44.102 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:36:44.102 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:36:44.102 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:36:44.102 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:36:44.102 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:36:44.102 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:36:44.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:36:44.853 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:49:58 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:36:45.002 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:36:45.003 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:36:45.003 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:36:45.003 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:36:45.003 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:36:45.003 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:36:45.003 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:36:45.003 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:36:45.003 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:36:45.003 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:36:45.004 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:36:45.004 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 99 M 2024-03-25T19:36:45.004 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 449 M 2024-03-25T19:36:45.004 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:38:00.293 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 99 MB 01:15 2024-03-25T19:38:00.294 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:00.294 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.3 MB/s | 99 MB 01:15 2024-03-25T19:38:00.294 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:38:00.298 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:38:00.298 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:38:00.907 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:38:00.907 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:38:11.153 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:38:11.257 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:38:14.295 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:38:14.562 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:38:14.562 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:38:14.562 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:38:14.562 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:14.562 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:38:14.563 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:38:14.755 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:38:15.317 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:51:29 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:38:15.465 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:38:15.466 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:38:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:38:15.466 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:38:15.466 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:38:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:38:15.466 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:38:15.466 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:38:15.466 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:38:15.466 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:38:15.467 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:38:15.467 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 287 M 2024-03-25T19:38:15.467 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 1.2 G 2024-03-25T19:38:15.467 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:38:53.383 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.9 MB/s | 287 MB 02:30 2024-03-25T19:38:53.384 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:53.385 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.9 MB/s | 287 MB 02:30 2024-03-25T19:38:53.385 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:38:53.389 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:38:53.389 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:38:55.098 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:38:55.098 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:39:11.348 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 5.1 MB/s | 287 MB 00:55 2024-03-25T19:39:11.349 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:11.349 INFO:teuthology.orchestra.run.smithi098.stdout:Total 5.1 MB/s | 287 MB 00:55 2024-03-25T19:39:11.350 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:39:11.353 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:39:11.354 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:39:13.059 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:39:13.060 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:39:24.658 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:39:25.273 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:30.611 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:30.876 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:30.876 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:30.876 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:39:30.876 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:30.876 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:30.877 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:39:31.104 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:39:31.663 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:52:44 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:39:31.813 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:39:31.814 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:39:31.814 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:39:31.814 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:39:31.814 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:39:31.814 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:39:31.814 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:31.814 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:39:31.814 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:39:31.814 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:39:31.814 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:31.815 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 5.9 M 2024-03-25T19:39:31.815 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 28 M 2024-03-25T19:39:31.815 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:39:32.426 INFO:teuthology.orchestra.run.smithi081.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 9.7 MB/s | 5.9 MB 00:00 2024-03-25T19:39:32.426 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:32.427 INFO:teuthology.orchestra.run.smithi081.stdout:Total 9.7 MB/s | 5.9 MB 00:00 2024-03-25T19:39:32.427 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:39:32.430 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:39:32.430 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:39:32.498 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:39:32.498 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:39:33.231 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:39:33.358 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:33.983 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:34.237 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:34.237 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:34.237 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:39:34.237 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:34.237 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:34.237 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:39:34.439 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:39:35.000 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:52:48 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:39:35.151 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:39:35.151 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:39:35.152 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:39:35.152 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:39:35.152 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:39:35.152 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:39:35.152 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:35.152 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:39:35.152 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:39:35.152 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:39:35.152 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:35.152 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 109 k 2024-03-25T19:39:35.152 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 440 k 2024-03-25T19:39:35.152 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:39:35.362 INFO:teuthology.orchestra.run.smithi081.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 519 kB/s | 109 kB 00:00 2024-03-25T19:39:35.363 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:35.363 INFO:teuthology.orchestra.run.smithi081.stdout:Total 516 kB/s | 109 kB 00:00 2024-03-25T19:39:35.363 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:39:35.366 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:39:35.366 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:39:35.399 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:39:35.400 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:39:35.513 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:39:35.627 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:36.009 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:36.271 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:36.271 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:36.271 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:39:36.271 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:36.271 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:36.271 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:39:36.476 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:39:37.034 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:52:50 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:39:37.181 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:39:37.182 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:39:37.182 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:39:37.182 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:39:37.182 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:39:37.183 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:39:37.183 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:37.183 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:39:37.183 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:39:37.183 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:39:37.183 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:37.183 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 5.4 M 2024-03-25T19:39:37.183 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 25 M 2024-03-25T19:39:37.183 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:39:37.973 INFO:teuthology.orchestra.run.smithi081.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 6.8 MB/s | 5.4 MB 00:00 2024-03-25T19:39:37.974 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:37.974 INFO:teuthology.orchestra.run.smithi081.stdout:Total 6.8 MB/s | 5.4 MB 00:00 2024-03-25T19:39:37.975 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:39:37.978 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:39:37.978 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:39:38.043 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:39:38.044 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:39:38.717 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:39:38.822 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:39.367 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:39.627 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:39:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:39.627 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:39.628 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:39:39.831 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librados2-debuginfo 2024-03-25T19:39:40.396 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:52:53 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:39:40.547 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:39:40.548 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:39:40.548 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:39:40.548 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:39:40.548 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:39:40.548 INFO:teuthology.orchestra.run.smithi081.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:39:40.548 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:40.548 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:39:40.548 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:39:40.548 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:39:40.548 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:40.548 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 44 M 2024-03-25T19:39:40.548 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 253 M 2024-03-25T19:39:40.548 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:39:42.444 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:39:42.845 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:44.576 INFO:teuthology.orchestra.run.smithi081.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 11 MB/s | 44 MB 00:04 2024-03-25T19:39:44.577 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:44.578 INFO:teuthology.orchestra.run.smithi081.stdout:Total 11 MB/s | 44 MB 00:04 2024-03-25T19:39:44.578 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:39:44.582 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:39:44.582 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:39:44.883 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:39:44.883 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:39:49.954 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:39:50.050 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:50.569 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:50.848 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:50.849 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:39:50.849 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:39:50.849 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:50.849 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:39:50.849 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:39:51.078 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:39:51.642 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:53:05 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:39:51.790 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:39:51.791 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:39:51.791 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:39:51.792 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:39:51.792 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:39:51.792 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:39:51.792 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:39:51.792 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:39:51.792 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:39:51.792 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:39:51.792 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:39:51.792 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 5.9 M 2024-03-25T19:39:51.792 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 28 M 2024-03-25T19:39:51.792 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:39:51.990 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:52.262 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:52.262 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:52.262 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:39:52.262 INFO:teuthology.orchestra.run.smithi081.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:52.262 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:52.262 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:39:52.490 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:39:53.050 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:53:06 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:39:53.200 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:39:53.200 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:39:53.201 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:39:53.201 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:39:53.201 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:39:53.201 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:39:53.201 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:53.201 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:39:53.201 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:39:53.201 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:39:53.201 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:39:53.201 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 52 M 2024-03-25T19:39:53.201 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 308 M 2024-03-25T19:39:53.201 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:39:54.148 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 2.5 MB/s | 5.9 MB 00:02 2024-03-25T19:39:54.149 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:54.149 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.5 MB/s | 5.9 MB 00:02 2024-03-25T19:39:54.149 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:39:54.153 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:39:54.154 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:39:54.224 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:39:54.224 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:39:54.955 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:39:55.055 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:55.594 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:55.862 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:39:55.863 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:39:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:55.863 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:39:55.863 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:39:56.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:39:56.632 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:53:10 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:39:56.783 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:39:56.784 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:39:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:39:56.784 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:39:56.784 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:39:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:39:56.784 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:39:56.784 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:39:56.785 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:39:56.785 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:39:56.785 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:39:56.785 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 109 k 2024-03-25T19:39:56.785 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 440 k 2024-03-25T19:39:56.785 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:39:57.318 INFO:teuthology.orchestra.run.smithi098.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 204 kB/s | 109 kB 00:00 2024-03-25T19:39:57.318 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:57.319 INFO:teuthology.orchestra.run.smithi098.stdout:Total 203 kB/s | 109 kB 00:00 2024-03-25T19:39:57.319 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:39:57.322 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:39:57.322 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:39:57.357 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:39:57.358 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:39:57.470 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:39:57.566 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:57.951 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:39:58.211 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:39:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:58.211 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:39:58.211 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:39:58.398 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:39:58.957 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:53:12 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:39:59.107 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:39:59.108 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:39:59.108 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:39:59.108 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:39:59.108 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:39:59.108 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:39:59.108 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:39:59.109 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:39:59.109 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:39:59.109 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:39:59.109 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:39:59.109 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 5.4 M 2024-03-25T19:39:59.109 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 25 M 2024-03-25T19:39:59.109 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:40:00.087 INFO:teuthology.orchestra.run.smithi081.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 7.5 MB/s | 52 MB 00:06 2024-03-25T19:40:00.088 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:00.088 INFO:teuthology.orchestra.run.smithi081.stdout:Total 7.5 MB/s | 52 MB 00:06 2024-03-25T19:40:00.089 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:40:00.093 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:40:00.093 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:40:00.146 INFO:teuthology.orchestra.run.smithi098.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 5.2 MB/s | 5.4 MB 00:01 2024-03-25T19:40:00.147 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:00.147 INFO:teuthology.orchestra.run.smithi098.stdout:Total 5.2 MB/s | 5.4 MB 00:01 2024-03-25T19:40:00.147 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:40:00.151 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:40:00.151 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:40:00.219 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:40:00.220 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:40:00.439 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:40:00.440 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:40:00.894 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:40:01.003 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:40:01.581 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:40:01.836 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:40:01.836 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:01.836 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:40:01.836 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:01.836 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:01.836 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:40:02.036 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librados2-debuginfo 2024-03-25T19:40:02.596 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:53:16 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:40:02.746 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:40:02.746 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:02.746 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:40:02.747 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:02.747 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:40:02.747 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:40:02.747 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:02.747 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:40:02.747 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:02.747 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:40:02.747 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:02.747 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 44 M 2024-03-25T19:40:02.747 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 253 M 2024-03-25T19:40:02.747 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:40:06.457 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:40:06.543 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:08.756 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:09.033 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:09.033 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:09.033 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:40:09.033 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:09.033 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:09.033 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:40:09.210 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:40:09.767 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:53:22 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:40:09.915 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:40:09.916 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:09.916 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:40:09.916 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:09.917 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:40:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:40:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:09.917 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:40:09.917 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:09.917 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:40:09.917 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:09.917 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 615 k 2024-03-25T19:40:09.917 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 2.1 M 2024-03-25T19:40:09.917 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:40:10.382 INFO:teuthology.orchestra.run.smithi081.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 1.3 MB/s | 615 kB 00:00 2024-03-25T19:40:10.382 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:10.383 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.3 MB/s | 615 kB 00:00 2024-03-25T19:40:10.383 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:40:10.386 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:40:10.386 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:40:10.424 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:40:10.424 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:40:10.592 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:40:10.681 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:40:11.127 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:40:11.391 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:40:11.392 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:11.392 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:40:11.392 INFO:teuthology.orchestra.run.smithi081.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:11.392 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:11.392 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:40:11.588 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:40:11.632 INFO:teuthology.orchestra.run.smithi098.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 5.0 MB/s | 44 MB 00:08 2024-03-25T19:40:11.633 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:11.633 INFO:teuthology.orchestra.run.smithi098.stdout:Total 5.0 MB/s | 44 MB 00:08 2024-03-25T19:40:11.634 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:40:11.638 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:40:11.638 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:40:11.939 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:40:11.940 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:40:12.146 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:53:25 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:40:12.298 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:40:12.299 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:12.299 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:40:12.299 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:12.299 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:40:12.299 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:40:12.299 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:12.299 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:40:12.300 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:12.300 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:40:12.300 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:12.300 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 771 k 2024-03-25T19:40:12.300 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 2.7 M 2024-03-25T19:40:12.300 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:40:13.160 INFO:teuthology.orchestra.run.smithi081.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 896 kB/s | 771 kB 00:00 2024-03-25T19:40:13.160 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:13.161 INFO:teuthology.orchestra.run.smithi081.stdout:Total 894 kB/s | 771 kB 00:00 2024-03-25T19:40:13.161 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:40:13.165 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:40:13.165 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:40:13.206 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:40:13.207 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:40:13.392 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:40:13.494 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:40:13.901 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:40:14.157 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:40:14.158 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:14.158 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:40:14.158 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:14.158 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:14.158 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:40:14.349 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:40:14.907 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:53:27 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:40:15.058 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:40:15.058 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:15.059 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:40:15.059 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:15.059 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:40:15.059 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:40:15.059 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:15.059 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:40:15.059 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:15.059 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:40:15.059 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:15.059 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 1.1 M 2024-03-25T19:40:15.059 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 4.3 M 2024-03-25T19:40:15.059 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:40:15.530 INFO:teuthology.orchestra.run.smithi081.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 2.4 MB/s | 1.1 MB 00:00 2024-03-25T19:40:15.530 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:15.531 INFO:teuthology.orchestra.run.smithi081.stdout:Total 2.4 MB/s | 1.1 MB 00:00 2024-03-25T19:40:15.531 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:40:15.534 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:40:15.534 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:40:15.575 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:40:15.576 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:40:15.796 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:40:15.904 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:16.307 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:16.575 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:16.575 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:16.575 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:40:16.575 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:16.575 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:16.575 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:40:16.777 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:40:17.004 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:40:17.089 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:40:17.337 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:53:30 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:40:17.487 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:40:17.487 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:17.487 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:40:17.487 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:17.487 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:40:17.487 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:40:17.488 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:17.488 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:40:17.488 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:17.488 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:40:17.488 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:17.488 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 337 k 2024-03-25T19:40:17.488 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 1.0 M 2024-03-25T19:40:17.488 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:40:17.758 INFO:teuthology.orchestra.run.smithi081.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 1.2 MB/s | 337 kB 00:00 2024-03-25T19:40:17.759 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:17.759 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.2 MB/s | 337 kB 00:00 2024-03-25T19:40:17.759 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:40:17.762 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:40:17.762 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:40:17.798 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:40:17.798 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:40:17.937 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:40:18.048 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:18.458 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:18.715 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:18.715 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:18.715 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:40:18.715 INFO:teuthology.orchestra.run.smithi081.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:18.715 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:18.715 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:40:18.905 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:40:19.059 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:40:19.325 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:40:19.326 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:19.326 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:40:19.326 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:19.326 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:19.326 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:40:19.467 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:53:32 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:40:19.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:40:19.614 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:40:19.615 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:19.615 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:40:19.615 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:19.615 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:40:19.615 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:40:19.615 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:19.616 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:40:19.616 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:19.616 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:40:19.616 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:19.616 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 509 k 2024-03-25T19:40:19.616 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 2.5 M 2024-03-25T19:40:19.616 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:40:19.861 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 2.0 MB/s | 509 kB 00:00 2024-03-25T19:40:19.862 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:19.862 INFO:teuthology.orchestra.run.smithi081.stdout:Total 2.0 MB/s | 509 kB 00:00 2024-03-25T19:40:19.862 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:40:19.865 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:40:19.866 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:40:19.902 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:40:19.902 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:40:20.060 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:40:20.092 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:53:34 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:40:20.177 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:40:20.243 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:40:20.243 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:20.243 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:40:20.243 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:20.243 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:40:20.243 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:40:20.243 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:20.243 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:40:20.243 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:20.243 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:40:20.244 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:20.244 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 52 M 2024-03-25T19:40:20.244 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 308 M 2024-03-25T19:40:20.244 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:40:20.593 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:40:20.856 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:40:20.856 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:20.856 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:40:20.856 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:20.857 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:20.857 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:40:21.046 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:40:21.609 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:53:34 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:40:21.757 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:40:21.757 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:21.757 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:40:21.757 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:21.757 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:40:21.758 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:40:21.758 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:21.758 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:40:21.758 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:21.758 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:40:21.758 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:21.758 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 52 M 2024-03-25T19:40:21.758 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 298 M 2024-03-25T19:40:21.759 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:40:22.944 INFO:teuthology.orchestra.run.smithi098.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 19 MB/s | 52 MB 00:02 2024-03-25T19:40:22.944 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:22.945 INFO:teuthology.orchestra.run.smithi098.stdout:Total 19 MB/s | 52 MB 00:02 2024-03-25T19:40:22.945 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:40:22.949 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:40:22.949 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:40:23.293 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:40:23.294 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:40:26.973 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 9.9 MB/s | 52 MB 00:05 2024-03-25T19:40:26.974 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:26.974 INFO:teuthology.orchestra.run.smithi081.stdout:Total 9.9 MB/s | 52 MB 00:05 2024-03-25T19:40:26.974 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:40:26.978 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:40:26.979 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:40:27.316 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:40:27.316 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:40:29.312 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:40:29.418 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:31.678 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:31.932 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:31.932 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:31.932 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:40:31.932 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:31.933 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:31.933 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:40:32.115 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:40:32.677 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:53:46 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:40:32.827 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:40:32.827 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:32.827 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:40:32.827 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:32.828 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:40:32.828 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:40:32.828 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:32.828 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:40:32.828 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:32.828 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:40:32.828 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:32.828 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 615 k 2024-03-25T19:40:32.829 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 2.1 M 2024-03-25T19:40:32.829 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:40:33.234 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:40:33.329 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:40:33.580 INFO:teuthology.orchestra.run.smithi098.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 818 kB/s | 615 kB 00:00 2024-03-25T19:40:33.580 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:33.581 INFO:teuthology.orchestra.run.smithi098.stdout:Total 817 kB/s | 615 kB 00:00 2024-03-25T19:40:33.581 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:40:33.584 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:40:33.584 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:40:33.623 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:40:33.623 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:40:33.791 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:40:33.903 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:40:34.303 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:40:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:40:34.572 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:34.572 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:40:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:34.573 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:34.573 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:40:34.761 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:40:35.342 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:53:49 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:40:35.469 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:40:35.506 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:40:35.506 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:35.506 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:40:35.506 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:35.507 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:40:35.507 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:40:35.507 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:35.507 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:40:35.507 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:35.507 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:40:35.507 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:35.507 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 771 k 2024-03-25T19:40:35.508 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 2.7 M 2024-03-25T19:40:35.508 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:40:35.738 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:40:35.738 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:35.738 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:40:35.738 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:35.738 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:35.738 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:40:35.927 DEBUG:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:40:36.489 INFO:teuthology.orchestra.run.smithi081.stdout:Last metadata expiration check: 0:53:49 ago on Mon 25 Mar 2024 06:46:47 PM UTC. 2024-03-25T19:40:36.500 INFO:teuthology.orchestra.run.smithi098.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 777 kB/s | 771 kB 00:00 2024-03-25T19:40:36.500 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:36.501 INFO:teuthology.orchestra.run.smithi098.stdout:Total 776 kB/s | 771 kB 00:00 2024-03-25T19:40:36.501 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:40:36.505 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:40:36.505 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:40:36.545 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:40:36.545 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:40:36.639 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies resolved. 2024-03-25T19:40:36.640 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:36.640 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repo Size 2024-03-25T19:40:36.640 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:36.640 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2024-03-25T19:40:36.640 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:40:36.640 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:36.640 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2024-03-25T19:40:36.641 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2024-03-25T19:40:36.641 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2024-03-25T19:40:36.641 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:36.641 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 1.3 M 2024-03-25T19:40:36.641 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 5.5 M 2024-03-25T19:40:36.641 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading Packages: 2024-03-25T19:40:36.723 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:40:36.825 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:40:37.218 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:40:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:40:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:37.482 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:40:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:37.482 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:37.482 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:40:37.650 INFO:teuthology.orchestra.run.smithi081.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.2 MB/s | 1.3 MB 00:01 2024-03-25T19:40:37.650 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:37.651 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.2 MB/s | 1.3 MB 00:01 2024-03-25T19:40:37.651 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2024-03-25T19:40:37.655 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction check succeeded. 2024-03-25T19:40:37.655 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2024-03-25T19:40:37.681 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:40:37.699 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded. 2024-03-25T19:40:37.700 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2024-03-25T19:40:37.948 INFO:teuthology.orchestra.run.smithi081.stdout: Preparing : 1/1 2024-03-25T19:40:38.057 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:38.242 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:53:52 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:40:38.391 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:40:38.392 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:38.392 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:40:38.392 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:38.392 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:40:38.392 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:40:38.392 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:38.392 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:40:38.393 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:38.393 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:40:38.393 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:38.393 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 1.1 M 2024-03-25T19:40:38.393 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 4.3 M 2024-03-25T19:40:38.393 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:40:38.494 INFO:teuthology.orchestra.run.smithi081.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:38.745 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:38.745 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:38.745 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2024-03-25T19:40:38.746 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:38.746 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:40:38.746 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2024-03-25T19:40:38.862 INFO:teuthology.orchestra.run.smithi098.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 2.4 MB/s | 1.1 MB 00:00 2024-03-25T19:40:38.863 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:38.863 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.4 MB/s | 1.1 MB 00:00 2024-03-25T19:40:38.863 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:40:38.866 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:40:38.866 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:40:38.907 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:40:38.907 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:40:38.927 DEBUG:teuthology.parallel:result is None 2024-03-25T19:40:39.128 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:40:39.219 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:39.592 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:39.856 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:39.856 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:39.856 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:40:39.856 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:39.856 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:39.857 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:40:40.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:40:40.637 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:53:54 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:40:40.786 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:40:40.787 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:40.787 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:40:40.787 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:40.787 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:40:40.787 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:40:40.787 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:40.787 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:40:40.788 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:40.788 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:40:40.788 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:40.788 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 337 k 2024-03-25T19:40:40.788 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 1.0 M 2024-03-25T19:40:40.788 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:40:41.173 INFO:teuthology.orchestra.run.smithi098.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 875 kB/s | 337 kB 00:00 2024-03-25T19:40:41.174 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:41.174 INFO:teuthology.orchestra.run.smithi098.stdout:Total 872 kB/s | 337 kB 00:00 2024-03-25T19:40:41.174 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:40:41.178 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:40:41.178 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:40:41.213 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:40:41.213 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:40:41.345 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:40:41.447 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:41.820 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:42.079 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:42.080 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:42.080 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:40:42.080 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:42.080 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:42.080 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:40:42.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:40:42.867 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:53:56 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:40:43.015 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:43.017 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 509 k 2024-03-25T19:40:43.017 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 2.5 M 2024-03-25T19:40:43.017 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:40:43.268 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 2.0 MB/s | 509 kB 00:00 2024-03-25T19:40:43.269 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:43.269 INFO:teuthology.orchestra.run.smithi098.stdout:Total 2.0 MB/s | 509 kB 00:00 2024-03-25T19:40:43.269 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:40:43.273 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:40:43.273 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:40:43.310 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:40:43.310 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:40:43.469 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:40:43.567 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:40:43.957 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:40:44.222 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:40:44.222 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:44.222 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:40:44.222 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:44.222 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:44.222 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:40:44.399 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:40:44.957 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:53:58 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:40:45.105 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:40:45.106 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:40:45.106 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:45.106 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:40:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:40:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:45.106 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:40:45.106 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:45.106 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:40:45.106 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:45.107 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 52 M 2024-03-25T19:40:45.107 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 298 M 2024-03-25T19:40:45.107 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:40:49.097 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 13 MB/s | 52 MB 00:03 2024-03-25T19:40:49.098 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:49.099 INFO:teuthology.orchestra.run.smithi098.stdout:Total 13 MB/s | 52 MB 00:03 2024-03-25T19:40:49.099 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:40:49.102 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:40:49.102 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:40:49.440 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:40:49.440 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:40:55.359 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:40:55.446 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:40:57.581 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:40:57.853 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:40:57.853 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:57.853 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:40:57.854 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:57.854 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:57.854 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:40:58.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:40:58.593 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:54:12 ago on Mon 25 Mar 2024 06:46:46 PM UTC. 2024-03-25T19:40:58.741 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-25T19:40:58.742 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:58.742 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repo Size 2024-03-25T19:40:58.742 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:58.742 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-25T19:40:58.742 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:40:58.742 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:58.742 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-25T19:40:58.742 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-25T19:40:58.742 INFO:teuthology.orchestra.run.smithi098.stdout:Install 1 Package 2024-03-25T19:40:58.742 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:40:58.743 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 1.3 M 2024-03-25T19:40:58.743 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 5.5 M 2024-03-25T19:40:58.743 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-25T19:40:59.932 INFO:teuthology.orchestra.run.smithi098.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.1 MB/s | 1.3 MB 00:01 2024-03-25T19:40:59.932 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:59.933 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.1 MB/s | 1.3 MB 00:01 2024-03-25T19:40:59.933 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-25T19:40:59.937 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-25T19:40:59.937 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-25T19:40:59.982 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-25T19:40:59.983 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-25T19:41:00.231 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-25T19:41:00.339 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:41:00.723 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:41:00.994 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:41:00.994 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:00.994 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-25T19:41:00.994 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:41:00.994 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:00.995 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-25T19:41:01.204 DEBUG:teuthology.parallel:result is None 2024-03-25T19:41:01.204 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:41:01.351 DEBUG:teuthology.orchestra.run.smithi081:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:41:01.375 INFO:teuthology.orchestra.run.smithi081.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:41:01.376 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:41:01.376 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:41:01.378 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:41:01.592 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:41:01.613 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:41:01.614 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:41:01.614 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:41:01.615 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:41:01.616 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:41:01.616 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:41:01.647 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T19:41:01.647 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:41:01.681 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:41:01.681 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:41:01.681 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:41:01.717 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:41:01.786 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T19:41:01.786 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:41:01.814 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:41:01.882 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:41:01.882 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:41:01.882 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:41:01.911 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:41:01.979 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T19:41:01.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:41:02.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:41:02.075 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:41:02.075 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:41:02.075 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:41:02.104 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:41:02.172 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T19:41:02.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:41:02.199 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:41:02.266 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:41:02.347 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:41:02.348 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 777 /var/log/ceph 2024-03-25T19:41:02.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /var/log/ceph 2024-03-25T19:41:02.376 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:41:02.377 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:41:02.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:41:02.444 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:41:02.444 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:41:02.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:41:02.523 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:41:02.523 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:41:02.523 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621334', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621334', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi081.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCN07iZlXHTVUpCzBERZtT0MJipiUFcqZGMF7B9u5DuDBQr05X96yQDfh6RwCfxsDrBc/m6JwKQt4dYEYGHw30c=', 'smithi098.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAwY3yfX1UoEShtlLxQrA1uQ4Cv4BCM43c1vEnjfLjFwEVlcfA35mwkmImJeqzHFIeBGPMSMzTHJBoPWDpViQLY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:41:02.524 DEBUG:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:41:02.553 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:41:02.568 DEBUG:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:41:02.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:41:02.636 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:41:02.636 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:41:02.690 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:41:02.690 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_1 2024-03-25T19:41:02.745 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:41:02.745 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:41:02.745 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T19:41:02.745 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:41:02.746 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:41:02.746 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-03-25 18:53:14.546526046 +0000 2024-03-25T19:41:02.746 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-03-25 18:45:50.359912003 +0000 2024-03-25T19:41:02.746 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-03-25 18:45:50.359912003 +0000 2024-03-25T19:41:02.746 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-03-25 18:45:50.359912003 +0000 2024-03-25T19:41:02.746 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:41:02.810 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-03-25T19:41:02.811 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-03-25T19:41:02.811 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000260256 s, 2.0 MB/s 2024-03-25T19:41:02.812 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:41:02.868 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_2 2024-03-25T19:41:02.923 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:41:02.924 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:41:02.924 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T19:41:02.924 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:41:02.924 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:41:02.924 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-03-25 18:53:14.546526046 +0000 2024-03-25T19:41:02.924 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-03-25 18:45:50.693905261 +0000 2024-03-25T19:41:02.924 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-03-25 18:45:50.693905261 +0000 2024-03-25T19:41:02.924 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-03-25 18:45:50.693905261 +0000 2024-03-25T19:41:02.924 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:41:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-03-25T19:41:02.989 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-03-25T19:41:02.990 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.00021104 s, 2.4 MB/s 2024-03-25T19:41:02.991 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:41:03.047 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_3 2024-03-25T19:41:03.102 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:41:03.102 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:41:03.102 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T19:41:03.102 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:41:03.102 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:41:03.102 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-03-25 18:53:14.546526046 +0000 2024-03-25T19:41:03.102 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-03-25 18:45:51.034898376 +0000 2024-03-25T19:41:03.102 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-03-25 18:45:51.034898376 +0000 2024-03-25T19:41:03.102 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-03-25 18:45:51.034898376 +0000 2024-03-25T19:41:03.102 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:41:03.167 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-03-25T19:41:03.167 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-03-25T19:41:03.167 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000180175 s, 2.8 MB/s 2024-03-25T19:41:03.168 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:41:03.223 DEBUG:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_4 2024-03-25T19:41:03.277 INFO:teuthology.orchestra.run.smithi081.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:41:03.278 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:41:03.278 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T19:41:03.278 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:41:03.278 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:41:03.278 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2024-03-25 18:53:14.547526026 +0000 2024-03-25T19:41:03.278 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2024-03-25 18:45:51.387891250 +0000 2024-03-25T19:41:03.278 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2024-03-25 18:45:51.387891250 +0000 2024-03-25T19:41:03.278 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: 2024-03-25 18:45:51.387891250 +0000 2024-03-25T19:41:03.278 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:41:03.343 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2024-03-25T19:41:03.343 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2024-03-25T19:41:03.343 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes copied, 0.000200721 s, 2.6 MB/s 2024-03-25T19:41:03.344 DEBUG:teuthology.orchestra.run.smithi081:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:41:03.399 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:41:03.399 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T19:41:03.399 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:41:03.417 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:41:03.417 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-03-25T19:41:03.475 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:41:03.475 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:41:03.475 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T19:41:03.475 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:41:03.475 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:41:03.475 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-25 18:53:07.593688171 +0000 2024-03-25T19:41:03.475 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-25 18:45:49.482671182 +0000 2024-03-25T19:41:03.475 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-25 18:45:49.482671182 +0000 2024-03-25T19:41:03.475 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-25 18:45:49.482671182 +0000 2024-03-25T19:41:03.476 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:41:03.545 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-25T19:41:03.545 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-25T19:41:03.546 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000227046 s, 2.3 MB/s 2024-03-25T19:41:03.547 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:41:03.605 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-03-25T19:41:03.662 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:41:03.662 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:41:03.662 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T19:41:03.662 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:41:03.662 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:41:03.662 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-25 18:53:07.593688171 +0000 2024-03-25T19:41:03.662 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-25 18:45:49.824666409 +0000 2024-03-25T19:41:03.662 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-25 18:45:49.824666409 +0000 2024-03-25T19:41:03.662 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-25 18:45:49.824666409 +0000 2024-03-25T19:41:03.662 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:41:03.730 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-25T19:41:03.730 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-25T19:41:03.730 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000204134 s, 2.5 MB/s 2024-03-25T19:41:03.731 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:41:03.788 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-03-25T19:41:03.844 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:41:03.844 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:41:03.844 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T19:41:03.844 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:41:03.844 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:41:03.844 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-25 18:53:07.593688171 +0000 2024-03-25T19:41:03.845 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-25 18:45:50.165661651 +0000 2024-03-25T19:41:03.845 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-25 18:45:50.165661651 +0000 2024-03-25T19:41:03.845 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-25 18:45:50.165661651 +0000 2024-03-25T19:41:03.845 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:41:03.911 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-25T19:41:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-25T19:41:03.912 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000273998 s, 1.9 MB/s 2024-03-25T19:41:03.913 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:41:03.969 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-03-25T19:41:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:41:04.024 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:41:04.025 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T19:41:04.025 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:41:04.025 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:41:04.025 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-25 18:53:07.594688157 +0000 2024-03-25T19:41:04.025 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-25 18:45:50.500656976 +0000 2024-03-25T19:41:04.025 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-25 18:45:50.500656976 +0000 2024-03-25T19:41:04.025 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-25 18:45:50.500656976 +0000 2024-03-25T19:41:04.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:41:04.090 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-25T19:41:04.090 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-25T19:41:04.090 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.00019585 s, 2.6 MB/s 2024-03-25T19:41:04.091 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:41:04.147 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:41:04.147 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:41:04.147 INFO:tasks.ceph:Generating config... 2024-03-25T19:41:04.148 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:41:04.149 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:41:04.149 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:41:04.149 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:41:04.149 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:41:04.149 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:41:04.149 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:41:04.149 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:41:04.149 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:41:04.149 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:41:04.149 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:41:04.149 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:41:04.149 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:41:04.149 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:41:04.149 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:41:04.149 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:41:04.150 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:41:04.150 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:41:04.150 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:41:04.150 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:41:04.150 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:41:04.150 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:41:04.150 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:41:04.150 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:41:04.150 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:41:04.150 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:41:04.150 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:41:04.150 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:41:04.150 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:41:04.150 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:41:04.151 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:41:04.151 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:41:04.151 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:41:04.151 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:41:04.151 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:41:04.151 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:41:04.151 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:41:04.151 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:41:04.151 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:41:04.151 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:41:04.151 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:41:04.151 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:41:04.151 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:41:04.151 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:41:04.151 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:41:04.152 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:41:04.152 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:41:04.152 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:41:04.152 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:41:04.152 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:41:04.152 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:41:04.152 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:41:04.152 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:41:04.152 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:41:04.152 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:41:04.208 INFO:teuthology.orchestra.run.smithi081.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:41:04.211 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:41:04.256 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:41:04.283 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.81:6789]'), ('mon.c', '[v1:172.21.15.81:6790]'), ('mon.b', '[v1:172.21.15.98:6789]')] 2024-03-25T19:41:04.283 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.81:6789],[v1:172.21.15.81:6790],[v1:172.21.15.98:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:41:04.284 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:41:04.284 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:41:04.338 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.81:6789]' --addv c '[v1:172.21.15.81:6790]' --addv b '[v1:172.21.15.98:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:04.422 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:04.422 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool: generated fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:41:04.423 INFO:teuthology.orchestra.run.smithi081.stdout:setting min_mon_release = quincy 2024-03-25T19:41:04.423 INFO:teuthology.orchestra.run.smithi081.stdout:epoch 0 2024-03-25T19:41:04.423 INFO:teuthology.orchestra.run.smithi081.stdout:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:41:04.423 INFO:teuthology.orchestra.run.smithi081.stdout:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:41:04.423 INFO:teuthology.orchestra.run.smithi081.stdout:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:41:04.423 INFO:teuthology.orchestra.run.smithi081.stdout:min_mon_release 17 (quincy) 2024-03-25T19:41:04.423 INFO:teuthology.orchestra.run.smithi081.stdout:election_strategy: 3 2024-03-25T19:41:04.423 INFO:teuthology.orchestra.run.smithi081.stdout:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:41:04.423 INFO:teuthology.orchestra.run.smithi081.stdout:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:41:04.423 INFO:teuthology.orchestra.run.smithi081.stdout:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:41:04.423 INFO:teuthology.orchestra.run.smithi081.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:41:04.424 DEBUG:teuthology.orchestra.run.smithi081:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:41:04.438 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0b5243e3-4e87-434a-ac8b-84a1b1f413df... 2024-03-25T19:41:04.439 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:41:04.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:41:04.529 INFO:teuthology.orchestra.run.smithi081.stdout:[global] 2024-03-25T19:41:04.529 INFO:teuthology.orchestra.run.smithi081.stdout: chdir = "" 2024-03-25T19:41:04.529 INFO:teuthology.orchestra.run.smithi081.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:41:04.529 INFO:teuthology.orchestra.run.smithi081.stdout: auth supported = cephx 2024-03-25T19:41:04.529 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.529 INFO:teuthology.orchestra.run.smithi081.stdout: filestore xattr use omap = true 2024-03-25T19:41:04.529 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.529 INFO:teuthology.orchestra.run.smithi081.stdout: mon clock drift allowed = 1.000 2024-03-25T19:41:04.529 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.529 INFO:teuthology.orchestra.run.smithi081.stdout: osd crush chooseleaf type = 0 2024-03-25T19:41:04.529 INFO:teuthology.orchestra.run.smithi081.stdout: auth debug = true 2024-03-25T19:41:04.529 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.530 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on old message = true 2024-03-25T19:41:04.530 INFO:teuthology.orchestra.run.smithi081.stdout: ms die on bug = true 2024-03-25T19:41:04.530 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.530 INFO:teuthology.orchestra.run.smithi081.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:41:04.530 INFO:teuthology.orchestra.run.smithi081.stdout: mon pg warn max object skew = 0 2024-03-25T19:41:04.530 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.530 INFO:teuthology.orchestra.run.smithi081.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:41:04.530 INFO:teuthology.orchestra.run.smithi081.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:41:04.530 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.530 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default size = 2 2024-03-25T19:41:04.530 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.530 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow primary affinity = true 2024-03-25T19:41:04.530 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd allow pg remap = true 2024-03-25T19:41:04.530 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:41:04.531 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:41:04.531 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on no sortbitwise = false 2024-03-25T19:41:04.531 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:41:04.531 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on too few osds = false 2024-03-25T19:41:04.531 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:41:04.531 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:41:04.531 INFO:teuthology.orchestra.run.smithi081.stdout: mon_allow_pool_size_one = true 2024-03-25T19:41:04.531 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.531 INFO:teuthology.orchestra.run.smithi081.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:41:04.531 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.531 INFO:teuthology.orchestra.run.smithi081.stdout: osd default data pool replay window = 5 2024-03-25T19:41:04.531 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.531 INFO:teuthology.orchestra.run.smithi081.stdout: mon allow pool delete = true 2024-03-25T19:41:04.531 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.531 INFO:teuthology.orchestra.run.smithi081.stdout: mon cluster log file level = debug 2024-03-25T19:41:04.532 INFO:teuthology.orchestra.run.smithi081.stdout: debug asserts on shutdown = true 2024-03-25T19:41:04.532 INFO:teuthology.orchestra.run.smithi081.stdout: mon health detail to clog = false 2024-03-25T19:41:04.532 INFO:teuthology.orchestra.run.smithi081.stdout: mon host = "[v1:172.21.15.81:6789],[v1:172.21.15.81:6790],[v1:172.21.15.98:6789]" 2024-03-25T19:41:04.532 INFO:teuthology.orchestra.run.smithi081.stdout: mon client directed command retry = 5 2024-03-25T19:41:04.532 INFO:teuthology.orchestra.run.smithi081.stdout: mon election default strategy = 3 2024-03-25T19:41:04.532 INFO:teuthology.orchestra.run.smithi081.stdout: ms bind msgr2 = False 2024-03-25T19:41:04.532 INFO:teuthology.orchestra.run.smithi081.stdout: ms inject socket failures = 5000 2024-03-25T19:41:04.532 INFO:teuthology.orchestra.run.smithi081.stdout: ms type = async 2024-03-25T19:41:04.532 INFO:teuthology.orchestra.run.smithi081.stdout: osd heartbeat grace = 80 2024-03-25T19:41:04.532 INFO:teuthology.orchestra.run.smithi081.stdout: fsid = 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:41:04.532 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.532 INFO:teuthology.orchestra.run.smithi081.stdout:[osd] 2024-03-25T19:41:04.532 INFO:teuthology.orchestra.run.smithi081.stdout: osd journal size = 100 2024-03-25T19:41:04.532 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.532 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub load threshold = 5.0 2024-03-25T19:41:04.532 INFO:teuthology.orchestra.run.smithi081.stdout: osd scrub max interval = 600 2024-03-25T19:41:04.533 INFO:teuthology.orchestra.run.smithi081.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:41:04.533 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.533 INFO:teuthology.orchestra.run.smithi081.stdout: osd recover clone overlap = true 2024-03-25T19:41:04.533 INFO:teuthology.orchestra.run.smithi081.stdout: osd recovery max chunk = 1048576 2024-03-25T19:41:04.533 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.533 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug shutdown = true 2024-03-25T19:41:04.533 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug op order = true 2024-03-25T19:41:04.533 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug verify stray on activate = true 2024-03-25T19:41:04.533 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug trim objects = true 2024-03-25T19:41:04.533 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.533 INFO:teuthology.orchestra.run.smithi081.stdout: osd open classes on start = true 2024-03-25T19:41:04.533 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug pg log writeout = true 2024-03-25T19:41:04.533 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.533 INFO:teuthology.orchestra.run.smithi081.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:41:04.533 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.533 INFO:teuthology.orchestra.run.smithi081.stdout: osd map max advance = 10 2024-03-25T19:41:04.534 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.534 INFO:teuthology.orchestra.run.smithi081.stdout: journal zero on create = true 2024-03-25T19:41:04.534 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.534 INFO:teuthology.orchestra.run.smithi081.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:41:04.534 INFO:teuthology.orchestra.run.smithi081.stdout: filestore apply finisher threads = 3 2024-03-25T19:41:04.534 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.534 INFO:teuthology.orchestra.run.smithi081.stdout: bdev debug aio = true 2024-03-25T19:41:04.534 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug misdirected ops = true 2024-03-25T19:41:04.534 INFO:teuthology.orchestra.run.smithi081.stdout: bdev async discard = True 2024-03-25T19:41:04.534 INFO:teuthology.orchestra.run.smithi081.stdout: bdev enable discard = True 2024-03-25T19:41:04.534 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore allocator = stupid 2024-03-25T19:41:04.534 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore block size = 96636764160 2024-03-25T19:41:04.534 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore fsck on mount = True 2024-03-25T19:41:04.535 INFO:teuthology.orchestra.run.smithi081.stdout: bluestore zero block detection = True 2024-03-25T19:41:04.535 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluefs = 1 2024-03-25T19:41:04.535 INFO:teuthology.orchestra.run.smithi081.stdout: debug bluestore = 1 2024-03-25T19:41:04.535 INFO:teuthology.orchestra.run.smithi081.stdout: debug monc = 20 2024-03-25T19:41:04.535 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-03-25T19:41:04.535 INFO:teuthology.orchestra.run.smithi081.stdout: debug osd = 20 2024-03-25T19:41:04.535 INFO:teuthology.orchestra.run.smithi081.stdout: debug rocksdb = 10 2024-03-25T19:41:04.535 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:41:04.535 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd full ratio = 0.9 2024-03-25T19:41:04.535 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:41:04.535 INFO:teuthology.orchestra.run.smithi081.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:41:04.535 INFO:teuthology.orchestra.run.smithi081.stdout: osd class default list = * 2024-03-25T19:41:04.535 INFO:teuthology.orchestra.run.smithi081.stdout: osd class load list = * 2024-03-25T19:41:04.535 INFO:teuthology.orchestra.run.smithi081.stdout: osd client watch timeout = 120 2024-03-25T19:41:04.535 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug verify cached snaps = True 2024-03-25T19:41:04.536 INFO:teuthology.orchestra.run.smithi081.stdout: osd debug verify missing on start = True 2024-03-25T19:41:04.536 INFO:teuthology.orchestra.run.smithi081.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:41:04.536 INFO:teuthology.orchestra.run.smithi081.stdout: osd fast shutdown = False 2024-03-25T19:41:04.536 INFO:teuthology.orchestra.run.smithi081.stdout: osd max markdown count = 1000 2024-03-25T19:41:04.536 INFO:teuthology.orchestra.run.smithi081.stdout: osd mclock override recovery settings = True 2024-03-25T19:41:04.536 INFO:teuthology.orchestra.run.smithi081.stdout: osd objectstore = bluestore 2024-03-25T19:41:04.536 INFO:teuthology.orchestra.run.smithi081.stdout: osd op queue = debug_random 2024-03-25T19:41:04.536 INFO:teuthology.orchestra.run.smithi081.stdout: osd op queue cut off = debug_random 2024-03-25T19:41:04.536 INFO:teuthology.orchestra.run.smithi081.stdout: osd shutdown pgref assert = True 2024-03-25T19:41:04.536 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.536 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr] 2024-03-25T19:41:04.536 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-03-25T19:41:04.536 INFO:teuthology.orchestra.run.smithi081.stdout: debug mgr = 20 2024-03-25T19:41:04.536 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2024-03-25T19:41:04.536 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2024-03-25T19:41:04.536 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:41:04.537 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:41:04.537 INFO:teuthology.orchestra.run.smithi081.stdout: mgr/telemetry/nag = false 2024-03-25T19:41:04.537 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.537 INFO:teuthology.orchestra.run.smithi081.stdout:[mon] 2024-03-25T19:41:04.537 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-03-25T19:41:04.537 INFO:teuthology.orchestra.run.smithi081.stdout: debug mon = 20 2024-03-25T19:41:04.537 INFO:teuthology.orchestra.run.smithi081.stdout: debug paxos = 20 2024-03-25T19:41:04.537 INFO:teuthology.orchestra.run.smithi081.stdout: debug auth = 20 2024-03-25T19:41:04.537 INFO:teuthology.orchestra.run.smithi081.stdout: mon data avail warn = 5 2024-03-25T19:41:04.537 INFO:teuthology.orchestra.run.smithi081.stdout: mon mgr mkfs grace = 240 2024-03-25T19:41:04.537 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:41:04.537 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd reporter subtree level = osd 2024-03-25T19:41:04.538 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd prime pg temp = true 2024-03-25T19:41:04.538 INFO:teuthology.orchestra.run.smithi081.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:41:04.538 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.538 INFO:teuthology.orchestra.run.smithi081.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:41:04.538 INFO:teuthology.orchestra.run.smithi081.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:41:04.538 INFO:teuthology.orchestra.run.smithi081.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:41:04.538 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.538 INFO:teuthology.orchestra.run.smithi081.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:41:04.538 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:41:04.538 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:41:04.538 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.538 INFO:teuthology.orchestra.run.smithi081.stdout: # 1m isn't quite enough 2024-03-25T19:41:04.538 INFO:teuthology.orchestra.run.smithi081.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:41:04.539 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.539 INFO:teuthology.orchestra.run.smithi081.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:41:04.539 INFO:teuthology.orchestra.run.smithi081.stdout: mon min osdmap epochs = 50 2024-03-25T19:41:04.539 INFO:teuthology.orchestra.run.smithi081.stdout: mon osd crush smoke test = False 2024-03-25T19:41:04.539 INFO:teuthology.orchestra.run.smithi081.stdout: mon osdmap full prune interval = 2 2024-03-25T19:41:04.539 INFO:teuthology.orchestra.run.smithi081.stdout: mon osdmap full prune min = 15 2024-03-25T19:41:04.539 INFO:teuthology.orchestra.run.smithi081.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:41:04.539 INFO:teuthology.orchestra.run.smithi081.stdout: mon scrub interval = 300 2024-03-25T19:41:04.539 INFO:teuthology.orchestra.run.smithi081.stdout: mon warn on pool no app = False 2024-03-25T19:41:04.539 INFO:teuthology.orchestra.run.smithi081.stdout: paxos service trim min = 10 2024-03-25T19:41:04.539 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:41:04.539 INFO:teuthology.orchestra.run.smithi081.stdout:[client] 2024-03-25T19:41:04.539 INFO:teuthology.orchestra.run.smithi081.stdout: rgw cache enabled = true 2024-03-25T19:41:04.539 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable ops log = true 2024-03-25T19:41:04.539 INFO:teuthology.orchestra.run.smithi081.stdout: rgw enable usage log = true 2024-03-25T19:41:04.539 INFO:teuthology.orchestra.run.smithi081.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:41:04.539 INFO:teuthology.orchestra.run.smithi081.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:41:04.540 INFO:teuthology.orchestra.run.smithi081.stdout: debug monc = 20 2024-03-25T19:41:04.540 INFO:teuthology.orchestra.run.smithi081.stdout: debug ms = 1 2024-03-25T19:41:04.540 INFO:teuthology.orchestra.run.smithi081.stdout: debug objecter = 20 2024-03-25T19:41:04.540 INFO:teuthology.orchestra.run.smithi081.stdout: debug rados = 20 2024-03-25T19:41:04.540 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.a] 2024-03-25T19:41:04.540 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.c] 2024-03-25T19:41:04.540 INFO:teuthology.orchestra.run.smithi081.stdout:[mon.b] 2024-03-25T19:41:04.540 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-03-25T19:41:04.540 INFO:teuthology.orchestra.run.smithi098.stdout: chdir = "" 2024-03-25T19:41:04.540 INFO:teuthology.orchestra.run.smithi098.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:41:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: auth supported = cephx 2024-03-25T19:41:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: filestore xattr use omap = true 2024-03-25T19:41:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: mon clock drift allowed = 1.000 2024-03-25T19:41:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: osd crush chooseleaf type = 0 2024-03-25T19:41:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: auth debug = true 2024-03-25T19:41:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on old message = true 2024-03-25T19:41:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: ms die on bug = true 2024-03-25T19:41:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:41:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: mon pg warn max object skew = 0 2024-03-25T19:41:04.541 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:41:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:41:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default size = 2 2024-03-25T19:41:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow primary affinity = true 2024-03-25T19:41:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd allow pg remap = true 2024-03-25T19:41:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:41:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:41:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on no sortbitwise = false 2024-03-25T19:41:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:41:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on too few osds = false 2024-03-25T19:41:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:41:04.542 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:41:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: mon_allow_pool_size_one = true 2024-03-25T19:41:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:41:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: osd default data pool replay window = 5 2024-03-25T19:41:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: mon allow pool delete = true 2024-03-25T19:41:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: mon cluster log file level = debug 2024-03-25T19:41:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: debug asserts on shutdown = true 2024-03-25T19:41:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: mon health detail to clog = false 2024-03-25T19:41:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: mon host = "[v1:172.21.15.81:6789],[v1:172.21.15.81:6790],[v1:172.21.15.98:6789]" 2024-03-25T19:41:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: mon client directed command retry = 5 2024-03-25T19:41:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: mon election default strategy = 3 2024-03-25T19:41:04.543 INFO:teuthology.orchestra.run.smithi098.stdout: ms bind msgr2 = False 2024-03-25T19:41:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: ms inject socket failures = 5000 2024-03-25T19:41:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: ms type = async 2024-03-25T19:41:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: osd heartbeat grace = 80 2024-03-25T19:41:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:41:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.544 INFO:teuthology.orchestra.run.smithi098.stdout:[osd] 2024-03-25T19:41:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: osd journal size = 100 2024-03-25T19:41:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub load threshold = 5.0 2024-03-25T19:41:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: osd scrub max interval = 600 2024-03-25T19:41:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:41:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: osd recover clone overlap = true 2024-03-25T19:41:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: osd recovery max chunk = 1048576 2024-03-25T19:41:04.544 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug shutdown = true 2024-03-25T19:41:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug op order = true 2024-03-25T19:41:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify stray on activate = true 2024-03-25T19:41:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug trim objects = true 2024-03-25T19:41:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: osd open classes on start = true 2024-03-25T19:41:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug pg log writeout = true 2024-03-25T19:41:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:41:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: osd map max advance = 10 2024-03-25T19:41:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: journal zero on create = true 2024-03-25T19:41:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.545 INFO:teuthology.orchestra.run.smithi098.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:41:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: filestore apply finisher threads = 3 2024-03-25T19:41:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: bdev debug aio = true 2024-03-25T19:41:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug misdirected ops = true 2024-03-25T19:41:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: bdev async discard = True 2024-03-25T19:41:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: bdev enable discard = True 2024-03-25T19:41:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore allocator = stupid 2024-03-25T19:41:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore block size = 96636764160 2024-03-25T19:41:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore fsck on mount = True 2024-03-25T19:41:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: bluestore zero block detection = True 2024-03-25T19:41:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluefs = 1 2024-03-25T19:41:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: debug bluestore = 1 2024-03-25T19:41:04.546 INFO:teuthology.orchestra.run.smithi098.stdout: debug monc = 20 2024-03-25T19:41:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-25T19:41:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: debug osd = 20 2024-03-25T19:41:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: debug rocksdb = 10 2024-03-25T19:41:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:41:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd full ratio = 0.9 2024-03-25T19:41:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:41:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:41:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: osd class default list = * 2024-03-25T19:41:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: osd class load list = * 2024-03-25T19:41:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: osd client watch timeout = 120 2024-03-25T19:41:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify cached snaps = True 2024-03-25T19:41:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: osd debug verify missing on start = True 2024-03-25T19:41:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:41:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: osd fast shutdown = False 2024-03-25T19:41:04.547 INFO:teuthology.orchestra.run.smithi098.stdout: osd max markdown count = 1000 2024-03-25T19:41:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: osd mclock override recovery settings = True 2024-03-25T19:41:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: osd objectstore = bluestore 2024-03-25T19:41:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: osd op queue = debug_random 2024-03-25T19:41:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: osd op queue cut off = debug_random 2024-03-25T19:41:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: osd shutdown pgref assert = True 2024-03-25T19:41:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.548 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr] 2024-03-25T19:41:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-25T19:41:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: debug mgr = 20 2024-03-25T19:41:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-03-25T19:41:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-03-25T19:41:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:41:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:41:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: mgr/telemetry/nag = false 2024-03-25T19:41:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.549 INFO:teuthology.orchestra.run.smithi098.stdout:[mon] 2024-03-25T19:41:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-25T19:41:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: debug mon = 20 2024-03-25T19:41:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: debug paxos = 20 2024-03-25T19:41:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: debug auth = 20 2024-03-25T19:41:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: mon data avail warn = 5 2024-03-25T19:41:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: mon mgr mkfs grace = 240 2024-03-25T19:41:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:41:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd reporter subtree level = osd 2024-03-25T19:41:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd prime pg temp = true 2024-03-25T19:41:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:41:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:41:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:41:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:41:04.549 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:41:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:41:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:41:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: # 1m isn't quite enough 2024-03-25T19:41:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:41:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:41:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: mon min osdmap epochs = 50 2024-03-25T19:41:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: mon osd crush smoke test = False 2024-03-25T19:41:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune interval = 2 2024-03-25T19:41:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune min = 15 2024-03-25T19:41:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:41:04.550 INFO:teuthology.orchestra.run.smithi098.stdout: mon scrub interval = 300 2024-03-25T19:41:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: mon warn on pool no app = False 2024-03-25T19:41:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: paxos service trim min = 10 2024-03-25T19:41:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-25T19:41:04.551 INFO:teuthology.orchestra.run.smithi098.stdout:[client] 2024-03-25T19:41:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: rgw cache enabled = true 2024-03-25T19:41:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable ops log = true 2024-03-25T19:41:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: rgw enable usage log = true 2024-03-25T19:41:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:41:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:41:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: debug monc = 20 2024-03-25T19:41:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: debug ms = 1 2024-03-25T19:41:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: debug objecter = 20 2024-03-25T19:41:04.551 INFO:teuthology.orchestra.run.smithi098.stdout: debug rados = 20 2024-03-25T19:41:04.551 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.a] 2024-03-25T19:41:04.551 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.c] 2024-03-25T19:41:04.551 INFO:teuthology.orchestra.run.smithi098.stdout:[mon.b] 2024-03-25T19:41:04.553 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:41:04.554 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:41:04.609 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:41:04.609 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:41:04.609 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:41:04.624 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:41:04.624 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:41:04.679 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T19:41:04.679 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:41:04.679 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:41:04.679 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:41:04.758 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:41:04.758 DEBUG:teuthology.orchestra.run.smithi081:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:04.813 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T19:41:04.813 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T19:41:04.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:41:04.813 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:41:04.853 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T19:41:04.853 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:04.908 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:41:04.908 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:41:04.909 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:41:04.963 INFO:teuthology.orchestra.run.smithi081.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:41:04.965 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:41:05.037 INFO:teuthology.orchestra.run.smithi098.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:41:05.039 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:41:05.040 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:41:05.040 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:41:05.082 INFO:teuthology.orchestra.run.smithi081.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:41:05.097 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:41:05.139 INFO:teuthology.orchestra.run.smithi098.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:41:05.154 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:41:05.154 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi098.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:41:05.155 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:41:05.181 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:41:05.181 INFO:tasks.ceph:role: osd.0 2024-03-25T19:41:05.181 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T19:41:05.181 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:41:05.255 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:41:05.255 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:41:05.255 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:41:05.255 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:41:05.255 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:41:05.255 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:41:05.255 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:41:05.255 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:41:05.255 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:41:05.255 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:41:05.749 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-03-25T19:41:05.751 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-03-25T19:41:05.751 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:41:05.889 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:41:05.923 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:41:05.991 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:41:05.991 INFO:tasks.ceph:role: osd.1 2024-03-25T19:41:05.991 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T19:41:05.992 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:41:06.065 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:41:06.065 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:41:06.065 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:41:06.065 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:41:06.066 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:41:06.066 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:41:06.066 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:41:06.066 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:41:06.066 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:41:06.066 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:41:06.530 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-03-25T19:41:06.533 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-03-25T19:41:06.533 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:41:06.617 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:41:06.690 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:41:06.758 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:41:06.758 INFO:tasks.ceph:role: osd.2 2024-03-25T19:41:06.758 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T19:41:06.759 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:41:06.832 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:41:06.833 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:41:06.833 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:41:06.833 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:41:06.833 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:41:06.833 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:41:06.833 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:41:06.833 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:41:06.833 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:41:06.833 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:41:07.337 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-03-25T19:41:07.339 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-03-25T19:41:07.340 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:41:07.493 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:41:07.527 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:41:07.596 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:41:07.596 INFO:tasks.ceph:role: osd.3 2024-03-25T19:41:07.596 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T19:41:07.596 DEBUG:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:41:07.672 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:41:07.672 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:41:07.673 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:41:07.673 INFO:teuthology.orchestra.run.smithi081.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:41:07.673 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:41:07.673 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:41:07.673 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:41:07.673 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:41:07.673 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:41:07.673 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:41:08.186 INFO:teuthology.orchestra.run.smithi081.stdout:Discarding blocks...Done. 2024-03-25T19:41:08.188 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2024-03-25T19:41:08.189 DEBUG:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:41:08.301 DEBUG:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:41:08.335 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:08.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:08.443+0000 7f3455814640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:41:08.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:08.443+0000 7f3455814640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:41:08.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:08.443+0000 7f3455814640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:41:08.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:08.443+0000 7f3455814640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:41:08.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:08.443+0000 7f3455814640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:41:08.443 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:08.444+0000 7f3455814640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:41:11.288 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:41:11.317 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:11.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:11.406+0000 7f516be12640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:41:11.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:11.407+0000 7f516be12640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:41:11.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:11.407+0000 7f516be12640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:41:11.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:11.407+0000 7f516be12640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:41:11.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:11.407+0000 7f516be12640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:41:11.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:11.407+0000 7f516be12640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:41:14.246 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:41:14.277 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:14.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:14.368+0000 7f09ad938640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:41:14.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:14.368+0000 7f09ad938640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:41:14.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:14.368+0000 7f09ad938640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:41:14.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:14.368+0000 7f09ad938640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:41:14.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:14.368+0000 7f09ad938640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:41:14.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:14.369+0000 7f09ad938640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:41:17.211 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:41:17.242 DEBUG:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:17.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:17.335+0000 7f4e189fa640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:41:17.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:17.336+0000 7f4e189fa640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:41:17.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:17.336+0000 7f4e189fa640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:41:17.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:17.336+0000 7f4e189fa640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:41:17.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:17.336+0000 7f4e189fa640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:41:17.336 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:17.336+0000 7f4e189fa640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:41:20.178 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:41:20.207 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:41:20.239 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:41:20.239 INFO:tasks.ceph:role: osd.4 2024-03-25T19:41:20.239 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T19:41:20.239 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:41:20.317 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:41:20.317 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:41:20.317 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:41:20.317 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:41:20.317 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:41:20.317 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:41:20.317 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:41:20.317 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:41:20.317 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:41:20.317 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:41:20.806 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-25T19:41:20.808 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-25T19:41:20.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:41:20.945 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:41:20.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:41:21.047 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:41:21.048 INFO:tasks.ceph:role: osd.5 2024-03-25T19:41:21.048 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T19:41:21.048 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:41:21.122 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:41:21.122 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:41:21.122 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:41:21.122 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:41:21.122 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:41:21.122 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:41:21.122 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:41:21.122 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:41:21.122 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:41:21.122 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:41:21.586 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-25T19:41:21.588 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-25T19:41:21.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:41:21.674 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:41:21.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:41:21.776 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:41:21.776 INFO:tasks.ceph:role: osd.6 2024-03-25T19:41:21.776 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T19:41:21.776 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:41:21.851 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:41:21.851 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:41:21.852 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:41:21.852 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:41:21.852 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:41:21.852 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:41:21.852 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:41:21.852 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:41:21.852 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:41:21.852 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:41:22.339 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-25T19:41:22.341 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-25T19:41:22.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:41:22.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:41:22.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:41:22.605 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:41:22.605 INFO:tasks.ceph:role: osd.7 2024-03-25T19:41:22.605 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T19:41:22.605 DEBUG:teuthology.orchestra.run.smithi098:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:41:22.680 INFO:teuthology.orchestra.run.smithi098.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:41:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:41:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:41:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:41:22.680 INFO:teuthology.orchestra.run.smithi098.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:41:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:41:22.680 INFO:teuthology.orchestra.run.smithi098.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:41:22.680 INFO:teuthology.orchestra.run.smithi098.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:41:22.680 INFO:teuthology.orchestra.run.smithi098.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:41:22.681 INFO:teuthology.orchestra.run.smithi098.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:41:23.176 INFO:teuthology.orchestra.run.smithi098.stdout:Discarding blocks...Done. 2024-03-25T19:41:23.178 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi098.front.sepia.ceph.com -o noatime 2024-03-25T19:41:23.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:41:23.334 DEBUG:teuthology.orchestra.run.smithi098:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:41:23.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:23.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:23.477+0000 7f0e21015640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:41:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:23.478+0000 7f0e21015640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:41:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:23.478+0000 7f0e21015640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:41:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:23.478+0000 7f0e21015640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:41:23.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:23.478+0000 7f0e21015640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:41:23.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:23.479+0000 7f0e21015640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:41:26.325 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:41:26.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:26.444+0000 7f42f6b1b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:41:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:26.444+0000 7f42f6b1b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:41:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:26.444+0000 7f42f6b1b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:41:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:26.444+0000 7f42f6b1b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:41:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:26.444+0000 7f42f6b1b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:41:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:26.444+0000 7f42f6b1b640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:41:29.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:41:29.310 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:29.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:29.401+0000 7f54bfe23640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:41:29.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:29.401+0000 7f54bfe23640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:41:29.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:29.401+0000 7f54bfe23640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:41:29.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:29.401+0000 7f54bfe23640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:41:29.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:29.401+0000 7f54bfe23640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:41:29.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:29.402+0000 7f54bfe23640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:41:32.248 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:41:32.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:32.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:32.372+0000 7f196ca68640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:41:32.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:32.372+0000 7f196ca68640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:41:32.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:32.372+0000 7f196ca68640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:41:32.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:32.372+0000 7f196ca68640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:41:32.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:32.372+0000 7f196ca68640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:41:32.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:41:32.372+0000 7f196ca68640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:41:35.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:41:35.239 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:41:35.239 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:41:35.240 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:41:35.271 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:41:35.271 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:41:35.341 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:41:35.341 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:41:35.410 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:41:35.410 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:41:35.480 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:41:35.480 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:41:35.549 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T19:41:35.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:41:35.580 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T19:41:35.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:41:35.650 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T19:41:35.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:41:35.719 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T19:41:35.719 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:41:35.787 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T19:41:35.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:41:35.855 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:41:35.855 DEBUG:teuthology.orchestra.run.smithi081:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:41:35.872 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T19:41:35.872 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:41:35.910 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:41:35.910 DEBUG:teuthology.orchestra.run.smithi081:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:41:35.914 DEBUG:teuthology.orchestra.run.smithi098:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:41:35.953 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr.y] 2024-03-25T19:41:35.953 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBQ0wFmYaOAORAAuwKWItbJnQY/uHGM7hkMcA== 2024-03-25T19:41:35.954 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.0] 2024-03-25T19:41:35.954 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBU0wFm/V18GhAASJyqBfhbAXLr9rvt20L4eA== 2024-03-25T19:41:35.954 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.1] 2024-03-25T19:41:35.954 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBX0wFmT7NLGBAAh4XPha2CcpzCbi18sXzafg== 2024-03-25T19:41:35.954 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.2] 2024-03-25T19:41:35.954 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBa0wFmeoAEFhAAXKvAm34oq7zSX06g4RGK4g== 2024-03-25T19:41:35.954 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.3] 2024-03-25T19:41:35.954 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBd0wFmUOsSFBAA91z0b5oH87ecthi23e94jA== 2024-03-25T19:41:35.954 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr.x] 2024-03-25T19:41:35.954 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBR0wFmWrlPAhAA5ADAF7740Bq/Ikdg4bI6qw== 2024-03-25T19:41:35.954 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.4] 2024-03-25T19:41:35.954 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBj0wFmdEh9HBAAlG+VnZW6MB1hwp5lGih5qQ== 2024-03-25T19:41:35.955 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.5] 2024-03-25T19:41:35.955 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBm0wFm7DSGGhAAFboiTlHJCX3crR2pAT5vPA== 2024-03-25T19:41:35.955 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.6] 2024-03-25T19:41:35.955 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBp0wFmslv7FxAAcyaVCGArKtUL6P1rFFdfRQ== 2024-03-25T19:41:35.955 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.7] 2024-03-25T19:41:35.955 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBs0wFmR3Q7FhAAfCnHMqhCMl4R/B1n0kfYGA== 2024-03-25T19:41:35.955 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2024-03-25T19:41:35.955 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBR0wFmx3/9BBAAHUeB6qcafYFZDri3eDxNCQ== 2024-03-25T19:41:35.955 INFO:teuthology.orchestra.run.smithi081.stdout:[client.1] 2024-03-25T19:41:35.955 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQBR0wFmFp1oCBAAjVSQrFPX23gcIsdIrU6j6g== 2024-03-25T19:41:35.976 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.y] 2024-03-25T19:41:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBQ0wFmYaOAORAAuwKWItbJnQY/uHGM7hkMcA== 2024-03-25T19:41:35.976 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.0] 2024-03-25T19:41:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBU0wFm/V18GhAASJyqBfhbAXLr9rvt20L4eA== 2024-03-25T19:41:35.976 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.1] 2024-03-25T19:41:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBX0wFmT7NLGBAAh4XPha2CcpzCbi18sXzafg== 2024-03-25T19:41:35.976 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.2] 2024-03-25T19:41:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBa0wFmeoAEFhAAXKvAm34oq7zSX06g4RGK4g== 2024-03-25T19:41:35.976 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.3] 2024-03-25T19:41:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBd0wFmUOsSFBAA91z0b5oH87ecthi23e94jA== 2024-03-25T19:41:35.976 INFO:teuthology.orchestra.run.smithi098.stdout:[mgr.x] 2024-03-25T19:41:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBR0wFmWrlPAhAA5ADAF7740Bq/Ikdg4bI6qw== 2024-03-25T19:41:35.976 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.4] 2024-03-25T19:41:35.976 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBj0wFmdEh9HBAAlG+VnZW6MB1hwp5lGih5qQ== 2024-03-25T19:41:35.977 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.5] 2024-03-25T19:41:35.977 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBm0wFm7DSGGhAAFboiTlHJCX3crR2pAT5vPA== 2024-03-25T19:41:35.977 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.6] 2024-03-25T19:41:35.977 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBp0wFmslv7FxAAcyaVCGArKtUL6P1rFFdfRQ== 2024-03-25T19:41:35.977 INFO:teuthology.orchestra.run.smithi098.stdout:[osd.7] 2024-03-25T19:41:35.977 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBs0wFmR3Q7FhAAfCnHMqhCMl4R/B1n0kfYGA== 2024-03-25T19:41:35.977 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-03-25T19:41:35.977 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBR0wFmx3/9BBAAHUeB6qcafYFZDri3eDxNCQ== 2024-03-25T19:41:35.977 INFO:teuthology.orchestra.run.smithi098.stdout:[client.1] 2024-03-25T19:41:35.977 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBR0wFmFp1oCBAAjVSQrFPX23gcIsdIrU6j6g== 2024-03-25T19:41:35.978 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:41:35.998 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:41:36.071 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:36.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:36.125 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:36.128 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:36.179 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:36.181 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:36.233 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:36.236 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:36.288 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:41:36.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:41:36.341 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:36.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:36.395 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:36.397 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:36.449 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:36.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:36.502 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:36.504 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:36.555 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:41:36.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:41:36.611 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:41:36.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:41:36.668 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:41:36.668 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:41:36.694 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:41:36.790 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:41:36.817 DEBUG:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:41:36.884 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:41:36.981 DEBUG:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:41:37.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:41:37.034 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:41:37.132 DEBUG:teuthology.orchestra.run.smithi098:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:41:37.158 DEBUG:teuthology.orchestra.run.smithi081:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:37.160 DEBUG:teuthology.orchestra.run.smithi098:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:37.213 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:41:37.213 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:41:37.213 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:41:37.213 DEBUG:teuthology.orchestra.run.smithi081:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:41:37.216 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:41:37.216 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:41:37.216 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:41:37.216 DEBUG:teuthology.orchestra.run.smithi081:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:41:37.218 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:41:37.218 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:41:37.218 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:41:37.219 DEBUG:teuthology.orchestra.run.smithi098:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:41:37.254 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:41:37.255 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:41:37.255 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:41:37.255 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:41:37.258 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:41:37.258 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:41:37.258 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:41:37.260 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:41:37.260 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:41:37.260 DEBUG:teuthology.orchestra.run.smithi081:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:41:37.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.362+0000 7fedc4b79640 10 monclient: get_monmap_and_config 2024-03-25T19:41:37.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.362+0000 7fedc4b79640 10 monclient: build_initial_monmap 2024-03-25T19:41:37.362 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.362+0000 7fedc4b79640 10 monclient: monmap: 2024-03-25T19:41:37.362 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T19:41:37.362 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:41:37.362 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T19:41:37.363567+0000 2024-03-25T19:41:37.362 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T19:41:37.363567+0000 2024-03-25T19:41:37.362 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:37.362 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T19:41:37.362 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:41:37.362 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:41:37.363 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:41:37.363 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T19:41:37.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.363+0000 7fedc4b79640 1 Processor -- start 2024-03-25T19:41:37.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.363+0000 7fedc4b79640 1 -- start start 2024-03-25T19:41:37.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.363+0000 7fedc4b79640 10 monclient: init 2024-03-25T19:41:37.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.363+0000 7fedc4b79640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:37.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.363+0000 7fedc4b79640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:37.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.363+0000 7fedc4b79640 10 monclient(hunting): picked mon.noname-c con 0x7fedc00b4700 addr v1:172.21.15.98:6789/0 2024-03-25T19:41:37.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.363+0000 7fedc4b79640 10 monclient(hunting): picked mon.noname-b con 0x7fedc0142a20 addr v1:172.21.15.81:6790/0 2024-03-25T19:41:37.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.364+0000 7fedc4b79640 10 monclient(hunting): picked mon.noname-a con 0x7fedc00a96b0 addr v1:172.21.15.81:6789/0 2024-03-25T19:41:37.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.364+0000 7fedc4b79640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc00a9d60 con 0x7fedc00a96b0 2024-03-25T19:41:37.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.364+0000 7fedc4b79640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc013ff10 con 0x7fedc00b4700 2024-03-25T19:41:37.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.364+0000 7fedc4b79640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc00aa1f0 con 0x7fedc0142a20 2024-03-25T19:41:37.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.364+0000 7fedc4b79640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:37.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.364+0000 7fedc4b79640 10 monclient(hunting): authenticate will time out at 3932.445876s 2024-03-25T19:41:37.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.364+0000 7fedbed9d640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7fedc00a96b0 legacy=0x7fedc013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T19:41:37.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.364+0000 7fedb7fff640 1 -- >> v1:172.21.15.81:6790/0 conn(0x7fedc0142a20 legacy=0x7fedc013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6790/0 2024-03-25T19:41:37.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.364+0000 7fedbe59c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fedc00b4700 legacy=0x7fedc0142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T19:41:37.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.565+0000 7fedbed9d640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7fedc00a96b0 legacy=0x7fedc013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T19:41:37.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.565+0000 7fedb7fff640 1 -- >> v1:172.21.15.81:6790/0 conn(0x7fedc0142a20 legacy=0x7fedc013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6790/0 2024-03-25T19:41:37.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.565+0000 7fedbe59c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fedc00b4700 legacy=0x7fedc0142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T19:41:37.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.966+0000 7fedbed9d640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7fedc00a96b0 legacy=0x7fedc013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T19:41:37.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.966+0000 7fedb7fff640 1 -- >> v1:172.21.15.81:6790/0 conn(0x7fedc0142a20 legacy=0x7fedc013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6790/0 2024-03-25T19:41:37.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:37.966+0000 7fedbe59c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fedc00b4700 legacy=0x7fedc0142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T19:41:38.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:38.767+0000 7fedb7fff640 1 -- >> v1:172.21.15.81:6790/0 conn(0x7fedc0142a20 legacy=0x7fedc013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6790/0 2024-03-25T19:41:38.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:38.767+0000 7fedbed9d640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7fedc00a96b0 legacy=0x7fedc013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T19:41:38.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:38.767+0000 7fedbe59c640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fedc00b4700 legacy=0x7fedc0142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T19:41:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedbcd99640 10 monclient(hunting): tick 2024-03-25T19:41:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedbcd99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:41:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedbcd99640 1 monclient(hunting): continuing hunt 2024-03-25T19:41:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedbcd99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:41:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedbcd99640 1 -- >> v1:172.21.15.81:6790/0 conn(0x7fedc0142a20 legacy=0x7fedc013f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedbcd99640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fedc00b4700 legacy=0x7fedc0142310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedbcd99640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7fedc00a96b0 legacy=0x7fedc013f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedbcd99640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:40.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-b con 0x7fedc0142a20 addr v1:172.21.15.81:6790/0 2024-03-25T19:41:40.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-a con 0x7fedc00a96b0 addr v1:172.21.15.81:6789/0 2024-03-25T19:41:40.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-c con 0x7fedc00b4700 addr v1:172.21.15.98:6789/0 2024-03-25T19:41:40.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedbcd99640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc00b4ae0 con 0x7fedc00a96b0 2024-03-25T19:41:40.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedbe59c640 1 -- >> v1:172.21.15.81:6790/0 conn(0x7fedc0142a20 legacy=0x7feda00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6790/0 2024-03-25T19:41:40.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedbcd99640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc00a9d60 con 0x7fedc00b4700 2024-03-25T19:41:40.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedb7fff640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7fedc00a96b0 legacy=0x7feda0008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T19:41:40.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedbcd99640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc013ff10 con 0x7fedc0142a20 2024-03-25T19:41:40.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedbcd99640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:40.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.364+0000 7fedbed9d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fedc00b4700 legacy=0x7feda000c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T19:41:40.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.565+0000 7fedbe59c640 1 -- >> v1:172.21.15.81:6790/0 conn(0x7fedc0142a20 legacy=0x7feda00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6790/0 2024-03-25T19:41:40.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.565+0000 7fedb7fff640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7fedc00a96b0 legacy=0x7feda0008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T19:41:40.565 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.565+0000 7fedbed9d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fedc00b4700 legacy=0x7feda000c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T19:41:40.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.966+0000 7fedb7fff640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7fedc00a96b0 legacy=0x7feda0008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T19:41:40.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.966+0000 7fedbe59c640 1 -- >> v1:172.21.15.81:6790/0 conn(0x7fedc0142a20 legacy=0x7feda00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6790/0 2024-03-25T19:41:40.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:40.966+0000 7fedbed9d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fedc00b4700 legacy=0x7feda000c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T19:41:41.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:41.767+0000 7fedbe59c640 1 -- >> v1:172.21.15.81:6790/0 conn(0x7fedc0142a20 legacy=0x7feda00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6790/0 2024-03-25T19:41:41.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:41.767+0000 7fedb7fff640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7fedc00a96b0 legacy=0x7feda0008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T19:41:41.767 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:41.767+0000 7fedbed9d640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fedc00b4700 legacy=0x7feda000c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T19:41:43.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedbcd99640 10 monclient(hunting): tick 2024-03-25T19:41:43.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedbcd99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:41:43.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedbcd99640 1 monclient(hunting): continuing hunt 2024-03-25T19:41:43.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedbcd99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:41:43.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedbcd99640 1 -- >> v1:172.21.15.81:6790/0 conn(0x7fedc0142a20 legacy=0x7feda00056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:43.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedbcd99640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fedc00b4700 legacy=0x7feda000c790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:43.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedbcd99640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7fedc00a96b0 legacy=0x7feda0008ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:43.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedbcd99640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:43.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-b con 0x7fedc0142a20 addr v1:172.21.15.81:6790/0 2024-03-25T19:41:43.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-c con 0x7fedc00b4700 addr v1:172.21.15.98:6789/0 2024-03-25T19:41:43.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-a con 0x7fedc00a96b0 addr v1:172.21.15.81:6789/0 2024-03-25T19:41:43.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedbcd99640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc00aa1f0 con 0x7fedc00a96b0 2024-03-25T19:41:43.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedbcd99640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc00b4ae0 con 0x7fedc00b4700 2024-03-25T19:41:43.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedbcd99640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc00a9d60 con 0x7fedc0142a20 2024-03-25T19:41:43.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedbcd99640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:43.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedbe59c640 1 -- >> v1:172.21.15.81:6790/0 conn(0x7fedc0142a20 legacy=0x7feda000ffe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6790/0 2024-03-25T19:41:43.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedbed9d640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7fedc00a96b0 legacy=0x7feda0008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T19:41:43.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.364+0000 7fedb7fff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fedc00b4700 legacy=0x7feda00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T19:41:43.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.565+0000 7fedbe59c640 1 -- >> v1:172.21.15.81:6790/0 conn(0x7fedc0142a20 legacy=0x7feda000ffe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6790/0 2024-03-25T19:41:43.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.565+0000 7fedbed9d640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7fedc00a96b0 legacy=0x7feda0008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T19:41:43.564 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.565+0000 7fedb7fff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fedc00b4700 legacy=0x7feda00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T19:41:43.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.966+0000 7fedbed9d640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7fedc00a96b0 legacy=0x7feda0008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T19:41:43.965 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.966+0000 7fedbe59c640 1 -- >> v1:172.21.15.81:6790/0 conn(0x7fedc0142a20 legacy=0x7feda000ffe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6790/0 2024-03-25T19:41:43.966 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:43.966+0000 7fedb7fff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fedc00b4700 legacy=0x7feda00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T19:41:44.229 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:00:06.863 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:44.455 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:00:07.097 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:44.456 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:00:07.098 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:44.535 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:00:07.169 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:44.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:44.767+0000 7fedbe59c640 1 -- >> v1:172.21.15.81:6790/0 conn(0x7fedc0142a20 legacy=0x7feda000ffe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6790/0 2024-03-25T19:41:44.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:44.767+0000 7fedbed9d640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7fedc00a96b0 legacy=0x7feda0008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.81:6789/0 2024-03-25T19:41:44.766 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:44.767+0000 7fedb7fff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fedc00b4700 legacy=0x7feda00056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.98:6789/0 2024-03-25T19:41:44.776 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:00:07.418 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:44.777 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:00:07.419 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:46.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:46.364+0000 7fedbcd99640 10 monclient(hunting): tick 2024-03-25T19:41:46.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:46.364+0000 7fedbcd99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:41:46.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:46.364+0000 7fedbcd99640 1 monclient(hunting): continuing hunt 2024-03-25T19:41:46.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:46.364+0000 7fedbcd99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:41:46.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:46.364+0000 7fedbcd99640 1 -- >> v1:172.21.15.81:6790/0 conn(0x7fedc0142a20 legacy=0x7feda000ffe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:46.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:46.364+0000 7fedbcd99640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7fedc00b4700 legacy=0x7feda00056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:46.364 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:46.364+0000 7fedbcd99640 1 -- >> v1:172.21.15.81:6789/0 conn(0x7fedc00a96b0 legacy=0x7feda0008ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:46.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:46.364+0000 7fedbcd99640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:46.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:46.364+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-a con 0x7fedc00a96b0 addr v1:172.21.15.81:6789/0 2024-03-25T19:41:46.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:46.364+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-b con 0x7fedc0142a20 addr v1:172.21.15.81:6790/0 2024-03-25T19:41:46.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:46.364+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-c con 0x7fedc00b4700 addr v1:172.21.15.98:6789/0 2024-03-25T19:41:46.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:46.364+0000 7fedbcd99640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc013ff10 con 0x7fedc00a96b0 2024-03-25T19:41:46.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:46.364+0000 7fedbcd99640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc00aa1f0 con 0x7fedc00b4700 2024-03-25T19:41:46.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:46.364+0000 7fedbcd99640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc00b4ae0 con 0x7fedc0142a20 2024-03-25T19:41:46.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:46.364+0000 7fedbcd99640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:46.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:46.413+0000 7fedb7fff640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fedc0142a20 0x7feda000ffe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:40574/0 (socket says 172.21.15.81:40574) 2024-03-25T19:41:46.413 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:46.413+0000 7fedb7fff640 1 -- 172.21.15.81:0/518106611 learned_addr learned my addr 172.21.15.81:0/518106611 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:41:49.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:49.364+0000 7fedbcd99640 10 monclient(hunting): tick 2024-03-25T19:41:49.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:49.364+0000 7fedbcd99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:41:49.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:49.364+0000 7fedbcd99640 1 monclient(hunting): continuing hunt 2024-03-25T19:41:49.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:49.364+0000 7fedbcd99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:41:49.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:49.364+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 >> v1:172.21.15.81:6790/0 conn(0x7fedc0142a20 legacy=0x7feda000ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:49.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:49.364+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 >> v1:172.21.15.98:6789/0 conn(0x7fedc00b4700 legacy=0x7feda00056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:49.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:49.364+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 >> v1:172.21.15.81:6789/0 conn(0x7fedc00a96b0 legacy=0x7feda000c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:49.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:49.364+0000 7fedbcd99640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:49.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:49.364+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-a con 0x7feda0008fa0 addr v1:172.21.15.81:6789/0 2024-03-25T19:41:49.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:49.364+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-b con 0x7feda00120a0 addr v1:172.21.15.81:6790/0 2024-03-25T19:41:49.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:49.364+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-c con 0x7feda0015bb0 addr v1:172.21.15.98:6789/0 2024-03-25T19:41:49.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:49.364+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc00a9d60 con 0x7feda0008fa0 2024-03-25T19:41:49.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:49.364+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda0010030 con 0x7feda0015bb0 2024-03-25T19:41:49.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:49.364+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda0010260 con 0x7feda00120a0 2024-03-25T19:41:49.368 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:49.364+0000 7fedbcd99640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:52.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:52.364+0000 7fedbcd99640 10 monclient(hunting): tick 2024-03-25T19:41:52.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:52.364+0000 7fedbcd99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:41:52.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:52.364+0000 7fedbcd99640 1 monclient(hunting): continuing hunt 2024-03-25T19:41:52.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:52.364+0000 7fedbcd99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:41:52.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:52.364+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 >> v1:172.21.15.81:6790/0 conn(0x7feda00120a0 legacy=0x7feda0014490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:52.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:52.364+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 >> v1:172.21.15.98:6789/0 conn(0x7feda0015bb0 legacy=0x7feda0017fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:52.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:52.364+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 >> v1:172.21.15.81:6789/0 conn(0x7feda0008fa0 legacy=0x7feda0010790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:52.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:52.364+0000 7fedbcd99640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:52.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:52.364+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-b con 0x7feda000c760 addr v1:172.21.15.81:6790/0 2024-03-25T19:41:52.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:52.364+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-a con 0x7feda0008fa0 addr v1:172.21.15.81:6789/0 2024-03-25T19:41:52.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:52.364+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-c con 0x7feda000d5d0 addr v1:172.21.15.98:6789/0 2024-03-25T19:41:52.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:52.364+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda00103f0 con 0x7feda0008fa0 2024-03-25T19:41:52.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:52.364+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda000d9b0 con 0x7feda000d5d0 2024-03-25T19:41:52.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:52.364+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda000db40 con 0x7feda000c760 2024-03-25T19:41:52.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:52.364+0000 7fedbcd99640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:55.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:55.365+0000 7fedbcd99640 10 monclient(hunting): tick 2024-03-25T19:41:55.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:55.365+0000 7fedbcd99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:41:55.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:55.365+0000 7fedbcd99640 1 monclient(hunting): continuing hunt 2024-03-25T19:41:55.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:55.365+0000 7fedbcd99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:41:55.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:55.365+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 >> v1:172.21.15.81:6790/0 conn(0x7feda000c760 legacy=0x7feda000cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:55.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:55.365+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 >> v1:172.21.15.98:6789/0 conn(0x7feda000d5d0 legacy=0x7feda0012480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:55.365+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 >> v1:172.21.15.81:6789/0 conn(0x7feda0008fa0 legacy=0x7feda0010790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:55.365+0000 7fedbcd99640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:55.365+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-c con 0x7feda0013bc0 addr v1:172.21.15.98:6789/0 2024-03-25T19:41:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:55.365+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-a con 0x7feda0015100 addr v1:172.21.15.81:6789/0 2024-03-25T19:41:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:55.365+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-b con 0x7feda000c760 addr v1:172.21.15.81:6790/0 2024-03-25T19:41:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:55.365+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda000dd20 con 0x7feda0015100 2024-03-25T19:41:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:55.365+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda0015680 con 0x7feda0013bc0 2024-03-25T19:41:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:55.365+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda00158b0 con 0x7feda000c760 2024-03-25T19:41:55.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:55.365+0000 7fedbcd99640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:58.365 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:58.365+0000 7fedbcd99640 10 monclient(hunting): tick 2024-03-25T19:41:58.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:58.365+0000 7fedbcd99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:41:58.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:58.365+0000 7fedbcd99640 1 monclient(hunting): continuing hunt 2024-03-25T19:41:58.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:58.365+0000 7fedbcd99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:41:58.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:58.365+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 >> v1:172.21.15.81:6790/0 conn(0x7feda000c760 legacy=0x7feda000cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:58.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:58.365+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 >> v1:172.21.15.98:6789/0 conn(0x7feda0013bc0 legacy=0x7feda0017fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:58.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:58.365+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 >> v1:172.21.15.81:6789/0 conn(0x7feda0015100 legacy=0x7feda00056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:58.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:58.365+0000 7fedbcd99640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:58.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:58.365+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-b con 0x7feda0013620 addr v1:172.21.15.81:6790/0 2024-03-25T19:41:58.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:58.365+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-c con 0x7feda0013bc0 addr v1:172.21.15.98:6789/0 2024-03-25T19:41:58.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:58.365+0000 7fedbcd99640 10 monclient(hunting): picked mon.noname-a con 0x7feda0015100 addr v1:172.21.15.81:6789/0 2024-03-25T19:41:58.366 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:58.366+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda0013490 con 0x7feda0015100 2024-03-25T19:41:58.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:58.366+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda0013a00 con 0x7feda0013bc0 2024-03-25T19:41:58.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:58.366+0000 7fedbcd99640 1 -- 172.21.15.81:0/518106611 --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda0018050 con 0x7feda0013620 2024-03-25T19:41:58.367 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:41:58.366+0000 7fedbcd99640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:00.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.375+0000 7fedbdd9b640 1 -- 172.21.15.81:0/518106611 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395559219 0 0) 0x7feda0013490 con 0x7feda0015100 2024-03-25T19:42:00.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.375+0000 7fedbdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:00.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.375+0000 7fedbdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:00.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.375+0000 7fedbdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:00.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.375+0000 7fedbdd9b640 10 monclient(hunting): my global_id is 4101 2024-03-25T19:42:00.375 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.375+0000 7fedbdd9b640 1 -- 172.21.15.81:0/518106611 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedac003520 con 0x7feda0015100 2024-03-25T19:42:00.406 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.407+0000 7fedbdd9b640 1 -- 172.21.15.81:0/518106611 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740575539 0 0) 0x7feda0013a00 con 0x7feda0013bc0 2024-03-25T19:42:00.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.407+0000 7fedbdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:00.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.407+0000 7fedbdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:00.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.407+0000 7fedbdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:00.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.407+0000 7fedbdd9b640 10 monclient(hunting): my global_id is 4105 2024-03-25T19:42:00.407 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.407+0000 7fedbdd9b640 1 -- 172.21.15.81:0/518106611 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda0013490 con 0x7feda0013bc0 2024-03-25T19:42:00.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.422+0000 7fedbdd9b640 1 -- 172.21.15.81:0/518106611 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2034732927 0 0) 0x7fedac003520 con 0x7feda0015100 2024-03-25T19:42:00.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.422+0000 7fedbdd9b640 1 -- 172.21.15.81:0/518106611 >> v1:172.21.15.81:6790/0 conn(0x7feda0013620 legacy=0x7feda0010770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.422+0000 7fedbdd9b640 1 -- 172.21.15.81:0/518106611 >> v1:172.21.15.98:6789/0 conn(0x7feda0013bc0 legacy=0x7feda00140c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.422+0000 7fedbdd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:42:00.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.422+0000 7fedbdd9b640 1 monclient: found mon.noname-a 2024-03-25T19:42:00.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.422+0000 7fedbdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:00.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.422+0000 7fedbdd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:42:00.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.422+0000 7fedbdd9b640 1 -- 172.21.15.81:0/518106611 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda0018280 con 0x7feda0015100 2024-03-25T19:42:00.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.422+0000 7fedbdd9b640 10 monclient: _finish_auth 0 2024-03-25T19:42:00.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.422+0000 7fedbdd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:42:00.422 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.422+0000 7fedbdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:00.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.422+0000 7fedbdd9b640 1 -- 172.21.15.81:0/518106611 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fedb80043b0 con 0x7feda0015100 2024-03-25T19:42:00.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.422+0000 7fedbdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:00.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.422+0000 7fedbdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:00.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.422+0000 7fedbdd9b640 10 monclient: dump: 2024-03-25T19:42:00.423 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T19:42:00.423 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:00.423 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:00.423 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:00.423 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:00.423 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:00.423 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:00.423 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:00.423 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:00.423 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T19:42:00.423 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.422+0000 7fedc4b79640 5 monclient: authenticate success, global_id 4101 2024-03-25T19:42:00.424 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.422+0000 7fedc4b79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:00.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.464+0000 7fedbdd9b640 1 -- 172.21.15.81:0/518106611 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedb8004510 con 0x7feda0015100 2024-03-25T19:42:00.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.464+0000 7fedbdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:00.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.464+0000 7fedbdd9b640 1 -- 172.21.15.81:0/518106611 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fedb8004960 con 0x7feda0015100 2024-03-25T19:42:00.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.464+0000 7fedc4b79640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:00.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.464+0000 7fedc4b79640 4 set_mon_vals no callback set 2024-03-25T19:42:00.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.464+0000 7fedbdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:00.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.464+0000 7fedbdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:00.464 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.464+0000 7fedbdd9b640 10 monclient: dump: 2024-03-25T19:42:00.464 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T19:42:00.464 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:00.464 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:00.464 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:00.465 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:00.465 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:00.465 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:00.465 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:00.465 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:00.465 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T19:42:00.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.465+0000 7fedc4b79640 10 monclient: shutdown 2024-03-25T19:42:00.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.465+0000 7fedc4b79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:00.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.465+0000 7fedc4b79640 1 -- 172.21.15.81:0/518106611 >> v1:172.21.15.81:6789/0 conn(0x7feda0015100 legacy=0x7feda000cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.465+0000 7fedc4b79640 1 -- 172.21.15.81:0/518106611 shutdown_connections 2024-03-25T19:42:00.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.465+0000 7fedc4b79640 1 -- 172.21.15.81:0/518106611 >> 172.21.15.81:0/518106611 conn(0x7fedc00668e0 msgr2=0x7fedc0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:00.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.465+0000 7fedc4b79640 1 -- 172.21.15.81:0/518106611 shutdown_connections 2024-03-25T19:42:00.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.465+0000 7fedc4b79640 1 -- 172.21.15.81:0/518106611 wait complete. 2024-03-25T19:42:00.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.466+0000 7fedc4b79640 10 monclient: build_initial_monmap 2024-03-25T19:42:00.468 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.466+0000 7fedc4b79640 10 monclient: monmap: 2024-03-25T19:42:00.468 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T19:42:00.468 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:00.468 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-25T19:42:00.468 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-25T19:42:00.469 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:00.469 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-25T19:42:00.469 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:00.469 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:00.469 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:00.469 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T19:42:00.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.466+0000 7fedc4b79640 1 librados: starting msgr at 2024-03-25T19:42:00.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.466+0000 7fedc4b79640 1 librados: starting objecter 2024-03-25T19:42:00.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.466+0000 7fedc4b79640 1 Processor -- start 2024-03-25T19:42:00.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.467+0000 7fedc4b79640 1 -- start start 2024-03-25T19:42:00.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.467+0000 7fedc4b79640 1 librados: setting wanted keys 2024-03-25T19:42:00.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.467+0000 7fedc4b79640 1 librados: calling monclient init 2024-03-25T19:42:00.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.467+0000 7fedc4b79640 10 monclient: init 2024-03-25T19:42:00.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.467+0000 7fedc4b79640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:00.469 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.467+0000 7fedc4b79640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:00.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.467+0000 7fedc4b79640 10 monclient(hunting): picked mon.noname-a con 0x7feda0013620 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:00.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.467+0000 7fedc4b79640 10 monclient(hunting): picked mon.noname-c con 0x7feda0013bc0 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:00.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.467+0000 7fedc4b79640 10 monclient(hunting): picked mon.noname-b con 0x7feda0015100 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:00.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.467+0000 7fedc4b79640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc01dbb40 con 0x7feda0013620 2024-03-25T19:42:00.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.467+0000 7fedc4b79640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc01db590 con 0x7feda0015100 2024-03-25T19:42:00.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.467+0000 7fedc4b79640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedc013b790 con 0x7feda0013bc0 2024-03-25T19:42:00.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.467+0000 7fedc4b79640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:00.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.467+0000 7fedc4b79640 10 monclient(hunting): authenticate will time out at 3955.549605s 2024-03-25T19:42:00.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.468+0000 7fedb7fff640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7feda0013bc0 0x7fedc011fa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:60246/0 (socket says 172.21.15.81:60246) 2024-03-25T19:42:00.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.468+0000 7fedb7fff640 1 -- 172.21.15.81:0/4108485319 learned_addr learned my addr 172.21.15.81:0/4108485319 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:00.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.468+0000 7fedb6ffd640 10 client.?.objecter ms_handle_connect 0x7feda0013bc0 2024-03-25T19:42:00.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.468+0000 7fedb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:00.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.468+0000 7fedb6ffd640 10 client.?.objecter ms_handle_connect 0x7feda0015100 2024-03-25T19:42:00.470 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.468+0000 7fedb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:00.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.520+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424122869 0 0) 0x7fedc01db590 con 0x7feda0015100 2024-03-25T19:42:00.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.520+0000 7fedb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:00.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.520+0000 7fedb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:00.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.520+0000 7fedb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:00.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.520+0000 7fedb6ffd640 10 monclient(hunting): my global_id is 4108 2024-03-25T19:42:00.520 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.520+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda0013a00 con 0x7feda0015100 2024-03-25T19:42:00.522 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.523+0000 7fedb6ffd640 10 client.?.objecter ms_handle_connect 0x7feda0013620 2024-03-25T19:42:00.523 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.523+0000 7fedb6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:00.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.544+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749755103 0 0) 0x7fedc013b790 con 0x7feda0013bc0 2024-03-25T19:42:00.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.544+0000 7fedb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:00.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.544+0000 7fedb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:00.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.544+0000 7fedb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:00.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.544+0000 7fedb6ffd640 10 monclient(hunting): my global_id is 4106 2024-03-25T19:42:00.543 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.544+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc01db590 con 0x7feda0013bc0 2024-03-25T19:42:00.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.548+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3755336419 0 0) 0x7feda0013a00 con 0x7feda0015100 2024-03-25T19:42:00.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.548+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedc013b790 con 0x7feda0015100 2024-03-25T19:42:00.548 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.548+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fedb8004820 con 0x7feda0015100 2024-03-25T19:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.548+0000 7fedb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.548+0000 7fedb6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.548+0000 7fedb6ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T19:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:00.549 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T19:42:00.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.548+0000 7fedb6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:00.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.549+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663329853 0 0) 0x7feda0018050 con 0x7feda0013620 2024-03-25T19:42:00.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.549+0000 7fedb6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:00.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.549+0000 7fedb6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:00.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.549+0000 7fedb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:00.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.549+0000 7fedb6ffd640 10 monclient(hunting): my global_id is 4110 2024-03-25T19:42:00.550 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.549+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda0013a00 con 0x7feda0013620 2024-03-25T19:42:00.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.569+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1293031812 0 0) 0x7feda0013a00 con 0x7feda0013620 2024-03-25T19:42:00.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.569+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feda0018050 con 0x7feda0013620 2024-03-25T19:42:00.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.569+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fedb0003ff0 con 0x7feda0013620 2024-03-25T19:42:00.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.569+0000 7fedb6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:00.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.569+0000 7fedb6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:00.569 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.569+0000 7fedb6ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:00.569 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T19:42:00.569 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:00.569 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:00.569 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:00.569 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:00.570 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:00.570 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:00.570 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:00.570 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:00.570 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T19:42:00.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2024305938 0 0) 0x7fedc013b790 con 0x7feda0015100 2024-03-25T19:42:00.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 >> v1:172.21.15.81:6790/0 conn(0x7feda0013bc0 legacy=0x7fedc011fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 >> v1:172.21.15.81:6789/0 conn(0x7feda0013620 legacy=0x7fedc00af0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:00.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedb6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:00.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedb6ffd640 1 monclient: found mon.b 2024-03-25T19:42:00.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:00.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedb6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:00.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc013b920 con 0x7feda0015100 2024-03-25T19:42:00.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedb6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:00.629 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedb6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:00.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:00.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedc4b79640 5 monclient: authenticate success, global_id 4108 2024-03-25T19:42:00.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedc4b79640 10 monclient: _renew_subs 2024-03-25T19:42:00.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedc4b79640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:00.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedc4b79640 1 -- 172.21.15.81:0/4108485319 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc00af7b0 con 0x7feda0015100 2024-03-25T19:42:00.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedc4b79640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:00.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedc4b79640 10 monclient: _renew_subs 2024-03-25T19:42:00.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedc4b79640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:00.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedc4b79640 1 -- 172.21.15.81:0/4108485319 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc01dae90 con 0x7feda0015100 2024-03-25T19:42:00.630 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.629+0000 7fedc4b79640 1 librados: init done 2024-03-25T19:42:00.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.630+0000 7fedc4b79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:00.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.630+0000 7fedc4b79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:00.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.630+0000 7fedc4b79640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:00.631 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.630+0000 7fedc4b79640 1 -- 172.21.15.81:0/4108485319 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc01dae90 con 0x7feda0015100 2024-03-25T19:42:00.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.882+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedb8004c30 con 0x7feda0015100 2024-03-25T19:42:00.881 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.882+0000 7fedb6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:00.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.882+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fedb8005040 con 0x7feda0015100 2024-03-25T19:42:00.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.882+0000 7fedb6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:00.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.882+0000 7fedb6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:00.882 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.882+0000 7fedb6ffd640 10 monclient: dump: 2024-03-25T19:42:00.882 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T19:42:00.882 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:00.882 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:00.882 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:00.882 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:00.882 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:00.882 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:00.882 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:00.882 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:00.882 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T19:42:00.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.882+0000 7fedbd59a640 4 set_mon_vals no callback set 2024-03-25T19:42:00.883 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.882+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fedb80055f0 con 0x7feda0015100 2024-03-25T19:42:00.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.884+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1844280568 0 0) 0x7fedb8006700 con 0x7feda0015100 2024-03-25T19:42:00.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.884+0000 7fedb6ffd640 10 client.4108.objecter ms_dispatch 0x7fedc0000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:42:00.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.884+0000 7fedb6ffd640 3 client.4108.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:42:00.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.884+0000 7fedb6ffd640 3 client.4108.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:42:00.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.885+0000 7fedb6ffd640 20 client.4108.objecter dump_active .. 0 homeless 2024-03-25T19:42:00.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.885+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fedb8005ad0 con 0x7feda0015100 2024-03-25T19:42:00.884 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.885+0000 7fedb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:00.885 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.885+0000 7fedb6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:00.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.952+0000 7fedc4b79640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:42:00.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.952+0000 7fedc4b79640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:42:00.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.952+0000 7fedc4b79640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:00.951 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:00.952+0000 7fedc4b79640 1 -- 172.21.15.81:0/4108485319 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fedc01dae90 con 0x7feda0015100 2024-03-25T19:42:01.085 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:01.086+0000 7f69b922b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:42:01.136 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:01.135+0000 7fbddaff1040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:42:01.155 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:01.155+0000 7f69b922b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:42:01.189 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:01.189+0000 7fbddaff1040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:42:01.399 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:01.400+0000 7fedb6ffd640 1 -- 172.21.15.81:0/4108485319 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fedb8005cf0 con 0x7feda0015100 2024-03-25T19:42:01.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:01.400+0000 7fedb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:42:01.400 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:01.400+0000 7fedb6ffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:42:01.400 INFO:teuthology.orchestra.run.smithi081.stderr:adjusted tunables profile to default 2024-03-25T19:42:01.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:01.400+0000 7fedc4b79640 10 librados: watch_flush enter 2024-03-25T19:42:01.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:01.401+0000 7fedc4b79640 10 librados: watch_flush exit 2024-03-25T19:42:01.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:01.401+0000 7fedc4b79640 20 client.4108.objecter shutdown clearing up homeless session... 2024-03-25T19:42:01.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:01.401+0000 7fedc4b79640 10 client.4108.objecter successfully canceled tick 2024-03-25T19:42:01.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:01.401+0000 7fedc4b79640 10 monclient: shutdown 2024-03-25T19:42:01.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:01.401+0000 7fedc4b79640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:01.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:01.401+0000 7fedc4b79640 1 -- 172.21.15.81:0/4108485319 >> v1:172.21.15.98:6789/0 conn(0x7feda0015100 legacy=0x7fedc0120150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:01.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:01.401+0000 7fedc4b79640 1 -- 172.21.15.81:0/4108485319 shutdown_connections 2024-03-25T19:42:01.401 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:01.401+0000 7fedc4b79640 1 -- 172.21.15.81:0/4108485319 >> 172.21.15.81:0/4108485319 conn(0x7fedc00668e0 msgr2=0x7fedc013d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:01.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:01.401+0000 7fedc4b79640 1 -- 172.21.15.81:0/4108485319 shutdown_connections 2024-03-25T19:42:01.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:01.401+0000 7fedc4b79640 1 -- 172.21.15.81:0/4108485319 wait complete. 2024-03-25T19:42:01.402 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:01.401+0000 7fedc4b79640 1 librados: shutdown 2024-03-25T19:42:01.411 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:42:01.411 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:42:01.412 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:42:01.412 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:42:01.440 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:42:01.440 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:42:01.508 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:42:01.509 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:42:01.575 DEBUG:teuthology.orchestra.run.smithi081:> set -ex 2024-03-25T19:42:01.575 DEBUG:teuthology.orchestra.run.smithi081:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:42:01.643 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T19:42:01.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:42:01.671 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T19:42:01.671 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:42:01.736 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:01.736+0000 7fbddaff1040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:42:01.739 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T19:42:01.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:42:01.792 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:01.792+0000 7fbddaff1040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:42:01.806 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-25T19:42:01.806 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:42:01.807 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:01.807+0000 7f69b922b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:42:01.839 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:01.839+0000 7fbddaff1040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:42:01.861 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:01.862+0000 7f69b922b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:42:01.864 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 67b3e439-45ad-438e-a3a0-a7eb0d3820c6 0 2024-03-25T19:42:01.906 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:01.906+0000 7fbddaff1040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:42:01.907 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:01.907+0000 7f69b922b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:42:01.950 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:01.951+0000 7fbddaff1040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:42:01.959 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:01.960+0000 7f69b922b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:42:02.001 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:02.001+0000 7f69b922b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:42:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.004+0000 7f5cf0096640 10 monclient: get_monmap_and_config 2024-03-25T19:42:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.004+0000 7f5cf0096640 10 monclient: build_initial_monmap 2024-03-25T19:42:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.004+0000 7f5cf0096640 10 monclient: monmap: 2024-03-25T19:42:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T19:42:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:42:02.005031+0000 2024-03-25T19:42:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:42:02.005031+0000 2024-03-25T19:42:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:42:02.004 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:42:02.004 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.004+0000 7f5cf0096640 1 Processor -- start 2024-03-25T19:42:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.004+0000 7f5cf0096640 1 -- start start 2024-03-25T19:42:02.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.004+0000 7f5cf0096640 10 monclient: init 2024-03-25T19:42:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.005+0000 7f5cf0096640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.005+0000 7f5cf0096640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.005+0000 7f5cf0096640 10 monclient(hunting): picked mon.noname-c con 0x7f5ce80b4700 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.005+0000 7f5cf0096640 10 monclient(hunting): picked mon.noname-a con 0x7f5ce8142a20 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.005+0000 7f5cf0096640 10 monclient(hunting): picked mon.noname-b con 0x7f5ce80a96b0 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.005+0000 7f5cf0096640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce80a9d60 con 0x7f5ce8142a20 2024-03-25T19:42:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.005+0000 7f5cf0096640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce813ff10 con 0x7f5ce80b4700 2024-03-25T19:42:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.005+0000 7f5cf0096640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce80aa1f0 con 0x7f5ce80a96b0 2024-03-25T19:42:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.005+0000 7f5cf0096640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.005+0000 7f5cf0096640 10 monclient(hunting): authenticate will time out at 3957.290163s 2024-03-25T19:42:02.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.006+0000 7f5cede32640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5ce80b4700 0x7f5ce8142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51762/0 (socket says 172.21.15.98:51762) 2024-03-25T19:42:02.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.006+0000 7f5cede32640 1 -- 172.21.15.98:0/274608546 learned_addr learned my addr 172.21.15.98:0/274608546 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T19:42:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.008+0000 7f5cece30640 1 -- 172.21.15.98:0/274608546 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453821564 0 0) 0x7f5ce813ff10 con 0x7f5ce80b4700 2024-03-25T19:42:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.008+0000 7f5cece30640 10 monclient(hunting): _negotiate 2024-03-25T19:42:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.008+0000 7f5cece30640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.008+0000 7f5cece30640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.008+0000 7f5cece30640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:42:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.008+0000 7f5cece30640 1 -- 172.21.15.98:0/274608546 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cd4003520 con 0x7f5ce80b4700 2024-03-25T19:42:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.008+0000 7f5cece30640 1 -- 172.21.15.98:0/274608546 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 365264249 0 0) 0x7f5ce80aa1f0 con 0x7f5ce80a96b0 2024-03-25T19:42:02.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.008+0000 7f5cece30640 10 monclient(hunting): _negotiate 2024-03-25T19:42:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.008+0000 7f5cece30640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.008+0000 7f5cece30640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.008+0000 7f5cece30640 10 monclient(hunting): my global_id is 4115 2024-03-25T19:42:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.008+0000 7f5cece30640 1 -- 172.21.15.98:0/274608546 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce813ff10 con 0x7f5ce80a96b0 2024-03-25T19:42:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.010+0000 7f5cece30640 1 -- 172.21.15.98:0/274608546 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839048252 0 0) 0x7f5ce80a9d60 con 0x7f5ce8142a20 2024-03-25T19:42:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.010+0000 7f5cece30640 10 monclient(hunting): _negotiate 2024-03-25T19:42:02.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.010+0000 7f5cece30640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.010+0000 7f5cece30640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.010+0000 7f5cece30640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:42:02.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.010+0000 7f5cece30640 1 -- 172.21.15.98:0/274608546 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce80aa1f0 con 0x7f5ce8142a20 2024-03-25T19:42:02.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.011+0000 7f5cece30640 1 -- 172.21.15.98:0/274608546 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1260264341 0 0) 0x7f5cd4003520 con 0x7f5ce80b4700 2024-03-25T19:42:02.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.011+0000 7f5cece30640 1 -- 172.21.15.98:0/274608546 >> v1:172.21.15.81:6790/0 conn(0x7f5ce80a96b0 legacy=0x7f5ce813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:02.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.011+0000 7f5cece30640 1 -- 172.21.15.98:0/274608546 >> v1:172.21.15.81:6789/0 conn(0x7f5ce8142a20 legacy=0x7f5ce813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:02.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.012+0000 7f5cece30640 10 monclient: _finish_hunting 0 2024-03-25T19:42:02.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.012+0000 7f5cece30640 1 monclient: found mon.noname-c 2024-03-25T19:42:02.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.012+0000 7f5cece30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:02.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.012+0000 7f5cece30640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:42:02.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.012+0000 7f5cece30640 1 -- 172.21.15.98:0/274608546 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce80b4ae0 con 0x7f5ce80b4700 2024-03-25T19:42:02.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.012+0000 7f5cece30640 10 monclient: _finish_auth 0 2024-03-25T19:42:02.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.012+0000 7f5cece30640 10 monclient: _check_auth_tickets 2024-03-25T19:42:02.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.012+0000 7f5cece30640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:02.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.012+0000 7f5cf0096640 5 monclient: authenticate success, global_id 4117 2024-03-25T19:42:02.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.012+0000 7f5cece30640 1 -- 172.21.15.98:0/274608546 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f5ce4001690 con 0x7f5ce80b4700 2024-03-25T19:42:02.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.012+0000 7f5cece30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:02.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.012+0000 7f5cece30640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:02.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.012+0000 7f5cece30640 10 monclient: dump: 2024-03-25T19:42:02.012 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:02.013 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.013 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.013 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.013 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:02.013 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.013 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:02.013 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:02.013 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:02.013 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.012+0000 7f5cf0096640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:02.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.014+0000 7f5cece30640 1 -- 172.21.15.98:0/274608546 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ce4001870 con 0x7f5ce80b4700 2024-03-25T19:42:02.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.014+0000 7f5cece30640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:02.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.014+0000 7f5cece30640 1 -- 172.21.15.98:0/274608546 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f5ce4001c80 con 0x7f5ce80b4700 2024-03-25T19:42:02.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.014+0000 7f5cf0096640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:02.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.014+0000 7f5cf0096640 4 set_mon_vals no callback set 2024-03-25T19:42:02.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.014+0000 7f5cece30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:02.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.014+0000 7f5cece30640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:02.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.014+0000 7f5cece30640 10 monclient: dump: 2024-03-25T19:42:02.015 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:02.015 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.015 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.015 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.015 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:02.015 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.015 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:02.015 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:02.015 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:02.015 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.014+0000 7f5cf0096640 10 monclient: shutdown 2024-03-25T19:42:02.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.014+0000 7f5cf0096640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:02.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.014+0000 7f5cf0096640 1 -- 172.21.15.98:0/274608546 >> v1:172.21.15.98:6789/0 conn(0x7f5ce80b4700 legacy=0x7f5ce8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.014+0000 7f5cf0096640 1 -- 172.21.15.98:0/274608546 shutdown_connections 2024-03-25T19:42:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.014+0000 7f5cf0096640 1 -- 172.21.15.98:0/274608546 >> 172.21.15.98:0/274608546 conn(0x7f5ce80668e0 msgr2=0x7f5ce8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.015+0000 7f5cf0096640 1 -- 172.21.15.98:0/274608546 shutdown_connections 2024-03-25T19:42:02.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.015+0000 7f5cf0096640 1 -- 172.21.15.98:0/274608546 wait complete. 2024-03-25T19:42:02.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.015+0000 7f5cf0096640 10 monclient: build_initial_monmap 2024-03-25T19:42:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.015+0000 7f5cf0096640 10 monclient: monmap: 2024-03-25T19:42:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T19:42:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-25T19:42:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-25T19:42:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-25T19:42:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:02.018 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.015+0000 7f5cf0096640 1 librados: starting msgr at 2024-03-25T19:42:02.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.015+0000 7f5cf0096640 1 librados: starting objecter 2024-03-25T19:42:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.015+0000 7f5cf0096640 1 Processor -- start 2024-03-25T19:42:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.016+0000 7f5cf0096640 1 -- start start 2024-03-25T19:42:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.016+0000 7f5cf0096640 1 librados: setting wanted keys 2024-03-25T19:42:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.016+0000 7f5cf0096640 1 librados: calling monclient init 2024-03-25T19:42:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.016+0000 7f5cf0096640 10 monclient: init 2024-03-25T19:42:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.016+0000 7f5cf0096640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.016+0000 7f5cf0096640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.016+0000 7f5cf0096640 10 monclient(hunting): picked mon.noname-b con 0x7f5ce80a96b0 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.016+0000 7f5cf0096640 10 monclient(hunting): picked mon.noname-a con 0x7f5ce80b4700 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.016+0000 7f5cf0096640 10 monclient(hunting): picked mon.noname-c con 0x7f5ce8142a20 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.016+0000 7f5cf0096640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce81d7380 con 0x7f5ce80b4700 2024-03-25T19:42:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.016+0000 7f5cf0096640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce81d4fe0 con 0x7f5ce80a96b0 2024-03-25T19:42:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.016+0000 7f5cf0096640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce81d8200 con 0x7f5ce8142a20 2024-03-25T19:42:02.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.016+0000 7f5cf0096640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.016+0000 7f5cf0096640 10 monclient(hunting): authenticate will time out at 3957.301292s 2024-03-25T19:42:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.017+0000 7f5cede32640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5ce80a96b0 0x7f5ce81d7af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51772/0 (socket says 172.21.15.98:51772) 2024-03-25T19:42:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.017+0000 7f5cede32640 1 -- 172.21.15.98:0/2361830425 learned_addr learned my addr 172.21.15.98:0/2361830425 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T19:42:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.017+0000 7f5cda7fc640 10 client.?.objecter ms_handle_connect 0x7f5ce80a96b0 2024-03-25T19:42:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.017+0000 7f5cda7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.017+0000 7f5cda7fc640 10 client.?.objecter ms_handle_connect 0x7f5ce80b4700 2024-03-25T19:42:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.017+0000 7f5cda7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.017+0000 7f5cda7fc640 10 client.?.objecter ms_handle_connect 0x7f5ce8142a20 2024-03-25T19:42:02.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.017+0000 7f5cda7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.019+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552045390 0 0) 0x7f5ce81d4fe0 con 0x7f5ce80a96b0 2024-03-25T19:42:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.019+0000 7f5cda7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.019+0000 7f5cda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.019+0000 7f5cda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.019+0000 7f5cda7fc640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:42:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.019+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ccc004eb0 con 0x7f5ce80a96b0 2024-03-25T19:42:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.019+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150957633 0 0) 0x7f5ce81d8200 con 0x7f5ce8142a20 2024-03-25T19:42:02.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.019+0000 7f5cda7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.019+0000 7f5cda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.019+0000 7f5cda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.019+0000 7f5cda7fc640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:42:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.019+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce81d4fe0 con 0x7f5ce8142a20 2024-03-25T19:42:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.020+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083646732 0 0) 0x7f5ce81d7380 con 0x7f5ce80b4700 2024-03-25T19:42:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.020+0000 7f5cda7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.020+0000 7f5cda7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.020+0000 7f5cda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.020+0000 7f5cda7fc640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:42:02.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.020+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ce81d8200 con 0x7f5ce80b4700 2024-03-25T19:42:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.023+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3210141404 0 0) 0x7f5ccc004eb0 con 0x7f5ce80a96b0 2024-03-25T19:42:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.023+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ce81d7380 con 0x7f5ce80a96b0 2024-03-25T19:42:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.023+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f5ce4001bc0 con 0x7f5ce80a96b0 2024-03-25T19:42:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.023+0000 7f5cda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.023+0000 7f5cda7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.023+0000 7f5cda7fc640 10 monclient(hunting): dump: 2024-03-25T19:42:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:02.024 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:02.025 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.024+0000 7f5cda7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.024+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3342518662 0 0) 0x7f5ce81d4fe0 con 0x7f5ce8142a20 2024-03-25T19:42:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.024+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ccc004eb0 con 0x7f5ce8142a20 2024-03-25T19:42:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.024+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f5ce0001a90 con 0x7f5ce8142a20 2024-03-25T19:42:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.024+0000 7f5cda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.024+0000 7f5cda7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.024+0000 7f5cda7fc640 10 monclient(hunting): dump: 2024-03-25T19:42:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.025 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:02.026 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:02.026 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.026+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1688812807 0 0) 0x7f5ce81d8200 con 0x7f5ce80b4700 2024-03-25T19:42:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.026+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ce81d4fe0 con 0x7f5ce80b4700 2024-03-25T19:42:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.026+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f5cdc0018f0 con 0x7f5ce80b4700 2024-03-25T19:42:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.026+0000 7f5cda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.026+0000 7f5cda7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.026+0000 7f5cda7fc640 10 monclient(hunting): dump: 2024-03-25T19:42:02.027 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:02.028 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2645461186 0 0) 0x7f5ce81d7380 con 0x7f5ce80a96b0 2024-03-25T19:42:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 >> v1:172.21.15.81:6790/0 conn(0x7f5ce8142a20 legacy=0x7f5ce81e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 >> v1:172.21.15.81:6789/0 conn(0x7f5ce80b4700 legacy=0x7f5ce80ac640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:02.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cda7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cda7fc640 1 monclient: found mon.b 2024-03-25T19:42:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cda7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce81d75d0 con 0x7f5ce80a96b0 2024-03-25T19:42:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cda7fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cda7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cf0096640 5 monclient: authenticate success, global_id 4120 2024-03-25T19:42:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cf0096640 10 monclient: _renew_subs 2024-03-25T19:42:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cf0096640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cf0096640 1 -- 172.21.15.98:0/2361830425 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce80acd50 con 0x7f5ce80a96b0 2024-03-25T19:42:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cf0096640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cf0096640 10 monclient: _renew_subs 2024-03-25T19:42:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cf0096640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:02.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cf0096640 1 -- 172.21.15.98:0/2361830425 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce80ac160 con 0x7f5ce80a96b0 2024-03-25T19:42:02.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.027+0000 7f5cf0096640 1 librados: init done 2024-03-25T19:42:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.028+0000 7f5cf0096640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.028+0000 7f5cf0096640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.028+0000 7f5cf0096640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.028+0000 7f5cf0096640 1 -- 172.21.15.98:0/2361830425 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce80add50 con 0x7f5ce80a96b0 2024-03-25T19:42:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.031+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ce4004bf0 con 0x7f5ce80a96b0 2024-03-25T19:42:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.031+0000 7f5cda7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.031+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f5ce4004f00 con 0x7f5ce80a96b0 2024-03-25T19:42:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.031+0000 7f5cda7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.031+0000 7f5cda7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.031+0000 7f5cda7fc640 10 monclient: dump: 2024-03-25T19:42:02.031 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:02.032 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.031+0000 7f5cdbfff640 4 set_mon_vals no callback set 2024-03-25T19:42:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.031+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5ce40053f0 con 0x7f5ce80a96b0 2024-03-25T19:42:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.031+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 206125934 0 0) 0x7f5ce4006750 con 0x7f5ce80a96b0 2024-03-25T19:42:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.031+0000 7f5cda7fc640 10 client.4120.objecter ms_dispatch 0x7f5ce8000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:42:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.031+0000 7f5cda7fc640 3 client.4120.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:42:02.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.031+0000 7f5cda7fc640 3 client.4120.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:42:02.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.031+0000 7f5cda7fc640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-25T19:42:02.071 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:02.071+0000 7fbddaff1040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:42:02.123 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:02.124+0000 7f69b922b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:42:02.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.125+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5ce4006ad0 con 0x7f5ce80a96b0 2024-03-25T19:42:02.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.125+0000 7f5cda7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:02.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.125+0000 7f5cda7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:02.143 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:02.143+0000 7fbddaff1040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:42:02.189 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:02.190+0000 7fbddaff1040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:42:02.193 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:02.193+0000 7f69b922b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:42:02.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.197+0000 7f5cf0096640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "67b3e439-45ad-438e-a3a0-a7eb0d3820c6", "id": 0}] 2024-03-25T19:42:02.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.197+0000 7f5cf0096640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "67b3e439-45ad-438e-a3a0-a7eb0d3820c6", "id": 0}] 2024-03-25T19:42:02.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.197+0000 7f5cf0096640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:02.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.197+0000 7f5cf0096640 1 -- 172.21.15.98:0/2361830425 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd new", "uuid": "67b3e439-45ad-438e-a3a0-a7eb0d3820c6", "id": 0} v 0) v1 -- 0x7f5ce804f730 con 0x7f5ce80a96b0 2024-03-25T19:42:02.243 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:02.243+0000 7fbddaff1040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:42:02.249 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:02.250+0000 7f69b922b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:42:02.291 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:02.291+0000 7fbddaff1040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:42:02.306 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:02.307+0000 7f69b922b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:42:02.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.348+0000 7f5cda7fc640 1 -- 172.21.15.98:0/2361830425 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "67b3e439-45ad-438e-a3a0-a7eb0d3820c6", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1342230774 0 732707515) 0x7f5ce40056a0 con 0x7f5ce80a96b0 2024-03-25T19:42:02.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.348+0000 7f5cda7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "67b3e439-45ad-438e-a3a0-a7eb0d3820c6", "id": 0}] 2024-03-25T19:42:02.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.348+0000 7f5cda7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:02.349 INFO:teuthology.orchestra.run.smithi098.stdout:0 2024-03-25T19:42:02.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.349+0000 7f5cf0096640 10 librados: watch_flush enter 2024-03-25T19:42:02.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.349+0000 7f5cf0096640 10 librados: watch_flush exit 2024-03-25T19:42:02.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.349+0000 7f5cf0096640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-25T19:42:02.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.349+0000 7f5cf0096640 10 client.4120.objecter successfully canceled tick 2024-03-25T19:42:02.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.349+0000 7f5cf0096640 10 monclient: shutdown 2024-03-25T19:42:02.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.349+0000 7f5cf0096640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:02.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.349+0000 7f5cf0096640 1 -- 172.21.15.98:0/2361830425 >> v1:172.21.15.98:6789/0 conn(0x7f5ce80a96b0 legacy=0x7f5ce81d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:02.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.349+0000 7f5cf0096640 1 -- 172.21.15.98:0/2361830425 shutdown_connections 2024-03-25T19:42:02.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.349+0000 7f5cf0096640 1 -- 172.21.15.98:0/2361830425 >> 172.21.15.98:0/2361830425 conn(0x7f5ce80668e0 msgr2=0x7f5ce80b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:02.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.349+0000 7f5cf0096640 1 -- 172.21.15.98:0/2361830425 shutdown_connections 2024-03-25T19:42:02.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.349+0000 7f5cf0096640 1 -- 172.21.15.98:0/2361830425 wait complete. 2024-03-25T19:42:02.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.350+0000 7f5cf0096640 1 librados: shutdown 2024-03-25T19:42:02.359 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:02.359+0000 7f69b922b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:42:02.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 4838341b-32d7-4f3d-8251-78eacaa81b32 1 2024-03-25T19:42:02.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.460+0000 7fed1732b640 10 monclient: get_monmap_and_config 2024-03-25T19:42:02.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.460+0000 7fed1732b640 10 monclient: build_initial_monmap 2024-03-25T19:42:02.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.460+0000 7fed1732b640 10 monclient: monmap: 2024-03-25T19:42:02.460 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T19:42:02.460 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.460 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:42:02.461151+0000 2024-03-25T19:42:02.460 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:42:02.461151+0000 2024-03-25T19:42:02.460 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:02.460 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.460 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:02.461 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:42:02.461 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:42:02.461 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.460+0000 7fed1732b640 1 Processor -- start 2024-03-25T19:42:02.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.461+0000 7fed1732b640 1 -- start start 2024-03-25T19:42:02.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.461+0000 7fed1732b640 10 monclient: init 2024-03-25T19:42:02.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.461+0000 7fed1732b640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:02.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.461+0000 7fed1732b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:02.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.461+0000 7fed1732b640 10 monclient(hunting): picked mon.noname-c con 0x7fed100b4700 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:02.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.461+0000 7fed1732b640 10 monclient(hunting): picked mon.noname-b con 0x7fed10142a20 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:02.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.461+0000 7fed1732b640 10 monclient(hunting): picked mon.noname-a con 0x7fed100a96b0 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:02.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.461+0000 7fed1732b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed100a9d60 con 0x7fed100a96b0 2024-03-25T19:42:02.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.461+0000 7fed1732b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed1013ff10 con 0x7fed100b4700 2024-03-25T19:42:02.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.461+0000 7fed1732b640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed100aa1f0 con 0x7fed10142a20 2024-03-25T19:42:02.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.461+0000 7fed1732b640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:02.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.461+0000 7fed1732b640 10 monclient(hunting): authenticate will time out at 3957.746387s 2024-03-25T19:42:02.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.463+0000 7fed148c6640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fed10142a20 0x7fed1013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.98:35554/0 (socket says 172.21.15.98:35554) 2024-03-25T19:42:02.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.463+0000 7fed148c6640 1 -- 172.21.15.98:0/2579402113 learned_addr learned my addr 172.21.15.98:0/2579402113 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T19:42:02.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.479+0000 7fed07fff640 1 -- 172.21.15.98:0/2579402113 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954772750 0 0) 0x7fed1013ff10 con 0x7fed100b4700 2024-03-25T19:42:02.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.479+0000 7fed07fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:02.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.479+0000 7fed07fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:02.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.479+0000 7fed07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:02.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.479+0000 7fed07fff640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:42:02.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.479+0000 7fed07fff640 1 -- 172.21.15.98:0/2579402113 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecfc003520 con 0x7fed100b4700 2024-03-25T19:42:02.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.483+0000 7fed07fff640 1 -- 172.21.15.98:0/2579402113 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2558233376 0 0) 0x7fecfc003520 con 0x7fed100b4700 2024-03-25T19:42:02.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.483+0000 7fed07fff640 1 -- 172.21.15.98:0/2579402113 >> v1:172.21.15.81:6790/0 conn(0x7fed10142a20 legacy=0x7fed1013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:02.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.483+0000 7fed07fff640 1 -- 172.21.15.98:0/2579402113 >> v1:172.21.15.81:6789/0 conn(0x7fed100a96b0 legacy=0x7fed1013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:02.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.483+0000 7fed07fff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:02.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.483+0000 7fed07fff640 1 monclient: found mon.noname-c 2024-03-25T19:42:02.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.483+0000 7fed07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.483+0000 7fed07fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:42:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.483+0000 7fed07fff640 1 -- 172.21.15.98:0/2579402113 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed100b4ae0 con 0x7fed100b4700 2024-03-25T19:42:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.483+0000 7fed07fff640 10 monclient: _finish_auth 0 2024-03-25T19:42:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.483+0000 7fed07fff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.483+0000 7fed07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.483+0000 7fed07fff640 1 -- 172.21.15.98:0/2579402113 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fed0c001690 con 0x7fed100b4700 2024-03-25T19:42:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.483+0000 7fed07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.483+0000 7fed07fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.483+0000 7fed07fff640 10 monclient: dump: 2024-03-25T19:42:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.484 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:02.485 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:02.485 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:02.485 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.483+0000 7fed1732b640 5 monclient: authenticate success, global_id 4123 2024-03-25T19:42:02.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.483+0000 7fed1732b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:02.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.485+0000 7fed07fff640 1 -- 172.21.15.98:0/2579402113 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed0c0017f0 con 0x7fed100b4700 2024-03-25T19:42:02.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.485+0000 7fed07fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:02.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.485+0000 7fed1732b640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:02.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.485+0000 7fed1732b640 4 set_mon_vals no callback set 2024-03-25T19:42:02.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.485+0000 7fed07fff640 1 -- 172.21.15.98:0/2579402113 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fed0c001c00 con 0x7fed100b4700 2024-03-25T19:42:02.487 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.485+0000 7fed07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.485+0000 7fed07fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.485+0000 7fed07fff640 10 monclient: dump: 2024-03-25T19:42:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:02.488 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.485+0000 7fed1732b640 10 monclient: shutdown 2024-03-25T19:42:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.485+0000 7fed1732b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.485+0000 7fed1732b640 1 -- 172.21.15.98:0/2579402113 >> v1:172.21.15.98:6789/0 conn(0x7fed100b4700 legacy=0x7fed10142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:02.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.485+0000 7fed1732b640 1 -- 172.21.15.98:0/2579402113 shutdown_connections 2024-03-25T19:42:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.485+0000 7fed1732b640 1 -- 172.21.15.98:0/2579402113 >> 172.21.15.98:0/2579402113 conn(0x7fed100668e0 msgr2=0x7fed10066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.485+0000 7fed1732b640 1 -- 172.21.15.98:0/2579402113 shutdown_connections 2024-03-25T19:42:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.485+0000 7fed1732b640 1 -- 172.21.15.98:0/2579402113 wait complete. 2024-03-25T19:42:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.486+0000 7fed1732b640 10 monclient: build_initial_monmap 2024-03-25T19:42:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.486+0000 7fed1732b640 10 monclient: monmap: 2024-03-25T19:42:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T19:42:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-25T19:42:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-25T19:42:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-25T19:42:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:02.489 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.486+0000 7fed1732b640 1 librados: starting msgr at 2024-03-25T19:42:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.486+0000 7fed1732b640 1 librados: starting objecter 2024-03-25T19:42:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.487+0000 7fed1732b640 1 Processor -- start 2024-03-25T19:42:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.487+0000 7fed1732b640 1 -- start start 2024-03-25T19:42:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.487+0000 7fed1732b640 1 librados: setting wanted keys 2024-03-25T19:42:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.487+0000 7fed1732b640 1 librados: calling monclient init 2024-03-25T19:42:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.487+0000 7fed1732b640 10 monclient: init 2024-03-25T19:42:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.487+0000 7fed1732b640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.487+0000 7fed1732b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.487+0000 7fed1732b640 10 monclient(hunting): picked mon.noname-c con 0x7fed100a96b0 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.487+0000 7fed1732b640 10 monclient(hunting): picked mon.noname-a con 0x7fed100b4700 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.487+0000 7fed1732b640 10 monclient(hunting): picked mon.noname-b con 0x7fed10142a20 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.487+0000 7fed1732b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed101391c0 con 0x7fed100b4700 2024-03-25T19:42:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.487+0000 7fed1732b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed101394a0 con 0x7fed10142a20 2024-03-25T19:42:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.487+0000 7fed1732b640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed10138090 con 0x7fed100a96b0 2024-03-25T19:42:02.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.487+0000 7fed1732b640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:02.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.487+0000 7fed1732b640 10 monclient(hunting): authenticate will time out at 3957.772427s 2024-03-25T19:42:02.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.488+0000 7fed158c8640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fed10142a20 0x7fed101e2580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51780/0 (socket says 172.21.15.98:51780) 2024-03-25T19:42:02.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.488+0000 7fed158c8640 1 -- 172.21.15.98:0/2855798221 learned_addr learned my addr 172.21.15.98:0/2855798221 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T19:42:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.488+0000 7fed05ffb640 10 client.?.objecter ms_handle_connect 0x7fed10142a20 2024-03-25T19:42:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.488+0000 7fed05ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.488+0000 7fed05ffb640 10 client.?.objecter ms_handle_connect 0x7fed100a96b0 2024-03-25T19:42:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.488+0000 7fed05ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.489+0000 7fed05ffb640 10 client.?.objecter ms_handle_connect 0x7fed100b4700 2024-03-25T19:42:02.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.489+0000 7fed05ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.490+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80382682 0 0) 0x7fed101394a0 con 0x7fed10142a20 2024-03-25T19:42:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.490+0000 7fed05ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.490+0000 7fed05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.490+0000 7fed05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.490+0000 7fed05ffb640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:42:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.490+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecf4004eb0 con 0x7fed10142a20 2024-03-25T19:42:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.491+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907849413 0 0) 0x7fed10138090 con 0x7fed100a96b0 2024-03-25T19:42:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.491+0000 7fed05ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.491+0000 7fed05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.491+0000 7fed05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.491+0000 7fed05ffb640 10 monclient(hunting): my global_id is 4121 2024-03-25T19:42:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.491+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed101394a0 con 0x7fed100a96b0 2024-03-25T19:42:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.492+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951540063 0 0) 0x7fed101391c0 con 0x7fed100b4700 2024-03-25T19:42:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.492+0000 7fed05ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.492+0000 7fed05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:02.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.492+0000 7fed05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.492+0000 7fed05ffb640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:42:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.492+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed10138090 con 0x7fed100b4700 2024-03-25T19:42:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.494+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1524426306 0 0) 0x7fecf4004eb0 con 0x7fed10142a20 2024-03-25T19:42:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.494+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed101391c0 con 0x7fed10142a20 2024-03-25T19:42:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.495+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fed08001dd0 con 0x7fed10142a20 2024-03-25T19:42:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.495+0000 7fed05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:02.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.495+0000 7fed05ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.495+0000 7fed05ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:02.495 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.495+0000 7fed05ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.497+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 571632564 0 0) 0x7fed101394a0 con 0x7fed100a96b0 2024-03-25T19:42:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.497+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecf4004eb0 con 0x7fed100a96b0 2024-03-25T19:42:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.497+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fed0c001b00 con 0x7fed100a96b0 2024-03-25T19:42:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.497+0000 7fed05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.497+0000 7fed05ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.497+0000 7fed05ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:02.497 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:02.498 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.497+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3139387716 0 0) 0x7fed10138090 con 0x7fed100b4700 2024-03-25T19:42:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.497+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed101394a0 con 0x7fed100b4700 2024-03-25T19:42:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.497+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fed000017c0 con 0x7fed100b4700 2024-03-25T19:42:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.497+0000 7fed05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.497+0000 7fed05ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.497+0000 7fed05ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.498 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:02.499 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:02.499 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:02.499 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1692625866 0 0) 0x7fed101391c0 con 0x7fed10142a20 2024-03-25T19:42:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 >> v1:172.21.15.81:6790/0 conn(0x7fed100a96b0 legacy=0x7fed101370b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 >> v1:172.21.15.81:6789/0 conn(0x7fed100b4700 legacy=0x7fed10134500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed05ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed05ffb640 1 monclient: found mon.b 2024-03-25T19:42:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed05ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:02.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed101385b0 con 0x7fed10142a20 2024-03-25T19:42:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed05ffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed05ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed1732b640 5 monclient: authenticate success, global_id 4126 2024-03-25T19:42:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed1732b640 10 monclient: _renew_subs 2024-03-25T19:42:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed1732b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed1732b640 1 -- 172.21.15.98:0/2855798221 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed10134c10 con 0x7fed10142a20 2024-03-25T19:42:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed1732b640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed1732b640 10 monclient: _renew_subs 2024-03-25T19:42:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed1732b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed1732b640 1 -- 172.21.15.98:0/2855798221 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed10137c70 con 0x7fed10142a20 2024-03-25T19:42:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.498+0000 7fed1732b640 1 librados: init done 2024-03-25T19:42:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.499+0000 7fed1732b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.499+0000 7fed1732b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.499+0000 7fed1732b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:02.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.499+0000 7fed1732b640 1 -- 172.21.15.98:0/2855798221 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed100b3690 con 0x7fed10142a20 2024-03-25T19:42:02.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.596+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed08001f30 con 0x7fed10142a20 2024-03-25T19:42:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.596+0000 7fed05ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.596+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fed08003de0 con 0x7fed10142a20 2024-03-25T19:42:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.596+0000 7fed05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.596+0000 7fed05ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.596+0000 7fed05ffb640 10 monclient: dump: 2024-03-25T19:42:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:02.597 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:02.598 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.596+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fed08004390 con 0x7fed10142a20 2024-03-25T19:42:02.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.596+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3317846707 0 0) 0x7fed08006900 con 0x7fed10142a20 2024-03-25T19:42:02.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.596+0000 7fed077fe640 4 set_mon_vals no callback set 2024-03-25T19:42:02.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.596+0000 7fed05ffb640 10 client.4126.objecter ms_dispatch 0x7fed10000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:42:02.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.596+0000 7fed05ffb640 3 client.4126.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:42:02.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.596+0000 7fed05ffb640 3 client.4126.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:42:02.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.596+0000 7fed05ffb640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-25T19:42:02.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.596+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fed08018de0 con 0x7fed10142a20 2024-03-25T19:42:02.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.596+0000 7fed05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:02.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.596+0000 7fed05ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:02.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.669+0000 7fed1732b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4838341b-32d7-4f3d-8251-78eacaa81b32", "id": 1}] 2024-03-25T19:42:02.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.669+0000 7fed1732b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4838341b-32d7-4f3d-8251-78eacaa81b32", "id": 1}] 2024-03-25T19:42:02.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.669+0000 7fed1732b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:02.669 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.669+0000 7fed1732b640 1 -- 172.21.15.98:0/2855798221 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4838341b-32d7-4f3d-8251-78eacaa81b32", "id": 1} v 0) v1 -- 0x7fed1004f730 con 0x7fed10142a20 2024-03-25T19:42:02.689 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:02.689+0000 7fbddaff1040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:42:02.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.746+0000 7fed05ffb640 1 -- 172.21.15.98:0/2855798221 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4838341b-32d7-4f3d-8251-78eacaa81b32", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 520469561 0 940483276) 0x7fed08005b40 con 0x7fed10142a20 2024-03-25T19:42:02.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.746+0000 7fed05ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4838341b-32d7-4f3d-8251-78eacaa81b32", "id": 1}] 2024-03-25T19:42:02.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.746+0000 7fed05ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:02.746 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-03-25T19:42:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.747+0000 7fed1732b640 10 librados: watch_flush enter 2024-03-25T19:42:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.747+0000 7fed1732b640 10 librados: watch_flush exit 2024-03-25T19:42:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.747+0000 7fed1732b640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-25T19:42:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.747+0000 7fed1732b640 10 client.4126.objecter successfully canceled tick 2024-03-25T19:42:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.747+0000 7fed1732b640 10 monclient: shutdown 2024-03-25T19:42:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.747+0000 7fed1732b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:02.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.747+0000 7fed1732b640 1 -- 172.21.15.98:0/2855798221 >> v1:172.21.15.98:6789/0 conn(0x7fed10142a20 legacy=0x7fed101e2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.747+0000 7fed1732b640 1 -- 172.21.15.98:0/2855798221 shutdown_connections 2024-03-25T19:42:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.747+0000 7fed1732b640 1 -- 172.21.15.98:0/2855798221 >> 172.21.15.98:0/2855798221 conn(0x7fed100668e0 msgr2=0x7fed100b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.747+0000 7fed1732b640 1 -- 172.21.15.98:0/2855798221 shutdown_connections 2024-03-25T19:42:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.747+0000 7fed1732b640 1 -- 172.21.15.98:0/2855798221 wait complete. 2024-03-25T19:42:02.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.747+0000 7fed1732b640 1 librados: shutdown 2024-03-25T19:42:02.756 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new ebc66313-ec95-428e-9fa9-559c14a21152 2 2024-03-25T19:42:02.767 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:02.767+0000 7f69b922b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:42:02.830 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:02.830+0000 7fbddaff1040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:42:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.855+0000 7f38298cf640 10 monclient: get_monmap_and_config 2024-03-25T19:42:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.855+0000 7f38298cf640 10 monclient: build_initial_monmap 2024-03-25T19:42:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.855+0000 7f38298cf640 10 monclient: monmap: 2024-03-25T19:42:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T19:42:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:42:02.856900+0000 2024-03-25T19:42:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:42:02.856900+0000 2024-03-25T19:42:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:42:02.856 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:42:02.857 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.856+0000 7f38298cf640 1 Processor -- start 2024-03-25T19:42:02.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.856+0000 7f38298cf640 1 -- start start 2024-03-25T19:42:02.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.856+0000 7f38298cf640 10 monclient: init 2024-03-25T19:42:02.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.857+0000 7f38298cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:02.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.857+0000 7f38298cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:02.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.857+0000 7f38298cf640 10 monclient(hunting): picked mon.noname-c con 0x7f38240b4700 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:02.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.857+0000 7f38298cf640 10 monclient(hunting): picked mon.noname-a con 0x7f3824142a20 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:02.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.857+0000 7f38298cf640 10 monclient(hunting): picked mon.noname-b con 0x7f38240a96b0 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:02.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.857+0000 7f38298cf640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38240a9d60 con 0x7f3824142a20 2024-03-25T19:42:02.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.857+0000 7f38298cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f382413ff10 con 0x7f38240b4700 2024-03-25T19:42:02.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.857+0000 7f38298cf640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38240aa1f0 con 0x7f38240a96b0 2024-03-25T19:42:02.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.857+0000 7f38298cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:02.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.857+0000 7f38298cf640 10 monclient(hunting): authenticate will time out at 3958.141909s 2024-03-25T19:42:02.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.857+0000 7f3822ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f38240b4700 0x7f3824142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51788/0 (socket says 172.21.15.98:51788) 2024-03-25T19:42:02.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.857+0000 7f3822ffd640 1 -- 172.21.15.98:0/2771892548 learned_addr learned my addr 172.21.15.98:0/2771892548 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T19:42:02.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 1 -- 172.21.15.98:0/2771892548 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702995218 0 0) 0x7f382413ff10 con 0x7f38240b4700 2024-03-25T19:42:02.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:02.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:02.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:02.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:42:02.860 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 1 -- 172.21.15.98:0/2771892548 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f380c003520 con 0x7f38240b4700 2024-03-25T19:42:02.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 1 -- 172.21.15.98:0/2771892548 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821031086 0 0) 0x7f38240aa1f0 con 0x7f38240a96b0 2024-03-25T19:42:02.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:02.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:02.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:02.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:42:02.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 1 -- 172.21.15.98:0/2771892548 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f382413ff10 con 0x7f38240a96b0 2024-03-25T19:42:02.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 1 -- 172.21.15.98:0/2771892548 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106877630 0 0) 0x7f38240a9d60 con 0x7f3824142a20 2024-03-25T19:42:02.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:02.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:02.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:02.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:42:02.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.860+0000 7f3821ffb640 1 -- 172.21.15.98:0/2771892548 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38240aa1f0 con 0x7f3824142a20 2024-03-25T19:42:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.863+0000 7f3821ffb640 1 -- 172.21.15.98:0/2771892548 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 941206288 0 0) 0x7f380c003520 con 0x7f38240b4700 2024-03-25T19:42:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.863+0000 7f3821ffb640 1 -- 172.21.15.98:0/2771892548 >> v1:172.21.15.81:6790/0 conn(0x7f38240a96b0 legacy=0x7f382413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.863+0000 7f3821ffb640 1 -- 172.21.15.98:0/2771892548 >> v1:172.21.15.81:6789/0 conn(0x7f3824142a20 legacy=0x7f382413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.863+0000 7f3821ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.863+0000 7f3821ffb640 1 monclient: found mon.noname-c 2024-03-25T19:42:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.863+0000 7f3821ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.863+0000 7f3821ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:42:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.863+0000 7f3821ffb640 1 -- 172.21.15.98:0/2771892548 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38240b4ae0 con 0x7f38240b4700 2024-03-25T19:42:02.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.863+0000 7f3821ffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.863+0000 7f3821ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.863+0000 7f3821ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.863+0000 7f3821ffb640 1 -- 172.21.15.98:0/2771892548 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3814001690 con 0x7f38240b4700 2024-03-25T19:42:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.863+0000 7f3821ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.863+0000 7f3821ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.863+0000 7f3821ffb640 10 monclient: dump: 2024-03-25T19:42:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:02.864 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:02.865 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.863+0000 7f38298cf640 5 monclient: authenticate success, global_id 4129 2024-03-25T19:42:02.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.863+0000 7f38298cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.864+0000 7f3821ffb640 1 -- 172.21.15.98:0/2771892548 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38140017f0 con 0x7f38240b4700 2024-03-25T19:42:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.864+0000 7f3821ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.864+0000 7f3821ffb640 1 -- 172.21.15.98:0/2771892548 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3814001c00 con 0x7f38240b4700 2024-03-25T19:42:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.864+0000 7f3821ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.864+0000 7f3821ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.864+0000 7f3821ffb640 10 monclient: dump: 2024-03-25T19:42:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.868 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:02.869 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.864+0000 7f38298cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.864+0000 7f38298cf640 4 set_mon_vals no callback set 2024-03-25T19:42:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.865+0000 7f38298cf640 10 monclient: shutdown 2024-03-25T19:42:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.865+0000 7f38298cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.865+0000 7f38298cf640 1 -- 172.21.15.98:0/2771892548 >> v1:172.21.15.98:6789/0 conn(0x7f38240b4700 legacy=0x7f3824142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.865+0000 7f38298cf640 1 -- 172.21.15.98:0/2771892548 shutdown_connections 2024-03-25T19:42:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.865+0000 7f38298cf640 1 -- 172.21.15.98:0/2771892548 >> 172.21.15.98:0/2771892548 conn(0x7f38240668e0 msgr2=0x7f3824066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.865+0000 7f38298cf640 1 -- 172.21.15.98:0/2771892548 shutdown_connections 2024-03-25T19:42:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.865+0000 7f38298cf640 1 -- 172.21.15.98:0/2771892548 wait complete. 2024-03-25T19:42:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.866+0000 7f38298cf640 10 monclient: build_initial_monmap 2024-03-25T19:42:02.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.866+0000 7f38298cf640 10 monclient: monmap: 2024-03-25T19:42:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T19:42:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-25T19:42:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-25T19:42:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-25T19:42:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:02.870 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.866+0000 7f38298cf640 1 librados: starting msgr at 2024-03-25T19:42:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.866+0000 7f38298cf640 1 librados: starting objecter 2024-03-25T19:42:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.866+0000 7f38298cf640 1 Processor -- start 2024-03-25T19:42:02.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.867+0000 7f38298cf640 1 -- start start 2024-03-25T19:42:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.867+0000 7f38298cf640 1 librados: setting wanted keys 2024-03-25T19:42:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.867+0000 7f38298cf640 1 librados: calling monclient init 2024-03-25T19:42:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.867+0000 7f38298cf640 10 monclient: init 2024-03-25T19:42:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.867+0000 7f38298cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.867+0000 7f38298cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.867+0000 7f38298cf640 10 monclient(hunting): picked mon.noname-b con 0x7f38240a96b0 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.867+0000 7f38298cf640 10 monclient(hunting): picked mon.noname-c con 0x7f38240b4700 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.867+0000 7f38298cf640 10 monclient(hunting): picked mon.noname-a con 0x7f3824142a20 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.867+0000 7f38298cf640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f382414eca0 con 0x7f3824142a20 2024-03-25T19:42:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.867+0000 7f38298cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38241d74d0 con 0x7f38240a96b0 2024-03-25T19:42:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.867+0000 7f38298cf640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38241d8350 con 0x7f38240b4700 2024-03-25T19:42:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.867+0000 7f38298cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.867+0000 7f38298cf640 10 monclient(hunting): authenticate will time out at 3958.152402s 2024-03-25T19:42:02.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.868+0000 7f38237fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f3824142a20 0x7f38241d4780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.98:44018/0 (socket says 172.21.15.98:44018) 2024-03-25T19:42:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.868+0000 7f38237fe640 1 -- 172.21.15.98:0/3194321321 learned_addr learned my addr 172.21.15.98:0/3194321321 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T19:42:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.868+0000 7f380bfff640 10 client.?.objecter ms_handle_connect 0x7f3824142a20 2024-03-25T19:42:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.868+0000 7f380bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.868+0000 7f380bfff640 10 client.?.objecter ms_handle_connect 0x7f38240b4700 2024-03-25T19:42:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.868+0000 7f380bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.868+0000 7f380bfff640 10 client.?.objecter ms_handle_connect 0x7f38240a96b0 2024-03-25T19:42:02.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.868+0000 7f380bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832055133 0 0) 0x7f382413ff10 con 0x7f3824142a20 2024-03-25T19:42:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:42:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3810004eb0 con 0x7f3824142a20 2024-03-25T19:42:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969520556 0 0) 0x7f38241d8350 con 0x7f38240b4700 2024-03-25T19:42:02.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:42:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f382413ff10 con 0x7f38240b4700 2024-03-25T19:42:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518626377 0 0) 0x7f38241d74d0 con 0x7f38240a96b0 2024-03-25T19:42:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:42:02.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.870+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38241d8350 con 0x7f38240a96b0 2024-03-25T19:42:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.875+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1423604853 0 0) 0x7f382413ff10 con 0x7f38240b4700 2024-03-25T19:42:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.875+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38241d74d0 con 0x7f38240b4700 2024-03-25T19:42:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.875+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3818001870 con 0x7f38240b4700 2024-03-25T19:42:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.875+0000 7f380bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.875+0000 7f380bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.875+0000 7f380bfff640 10 monclient(hunting): dump: 2024-03-25T19:42:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.875 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:02.876 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.875+0000 7f380bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.876+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 216861640 0 0) 0x7f38241d8350 con 0x7f38240a96b0 2024-03-25T19:42:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.876+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f382413ff10 con 0x7f38240a96b0 2024-03-25T19:42:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.876+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3814001b00 con 0x7f38240a96b0 2024-03-25T19:42:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.876+0000 7f380bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.876+0000 7f380bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.876+0000 7f380bfff640 10 monclient(hunting): dump: 2024-03-25T19:42:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:02.877 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.878 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:02.878 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:02.878 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:02.878 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.876+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 252623544 0 0) 0x7f3810004eb0 con 0x7f3824142a20 2024-03-25T19:42:02.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.876+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38241d8350 con 0x7f3824142a20 2024-03-25T19:42:02.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.876+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f381c001af0 con 0x7f3824142a20 2024-03-25T19:42:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.876+0000 7f380bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.876+0000 7f380bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.876+0000 7f380bfff640 10 monclient(hunting): dump: 2024-03-25T19:42:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:02.879 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.878+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2390129538 0 0) 0x7f38241d74d0 con 0x7f38240b4700 2024-03-25T19:42:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.878+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 >> v1:172.21.15.98:6789/0 conn(0x7f38240a96b0 legacy=0x7f38241d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.878+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 >> v1:172.21.15.81:6789/0 conn(0x7f3824142a20 legacy=0x7f38241d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.878+0000 7f380bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.878+0000 7f380bfff640 1 monclient: found mon.c 2024-03-25T19:42:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.878+0000 7f380bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.878+0000 7f380bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.878+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38240abd90 con 0x7f38240b4700 2024-03-25T19:42:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.878+0000 7f380bfff640 10 monclient: _finish_auth 0 2024-03-25T19:42:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.878+0000 7f380bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:02.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.878+0000 7f380bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:02.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.878+0000 7f38298cf640 5 monclient: authenticate success, global_id 4127 2024-03-25T19:42:02.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.878+0000 7f38298cf640 10 monclient: _renew_subs 2024-03-25T19:42:02.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.878+0000 7f38298cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:02.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.878+0000 7f38298cf640 1 -- 172.21.15.98:0/3194321321 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38241d7720 con 0x7f38240b4700 2024-03-25T19:42:02.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.879+0000 7f38298cf640 10 client.4127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:02.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.879+0000 7f38298cf640 10 monclient: _renew_subs 2024-03-25T19:42:02.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.879+0000 7f38298cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:02.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.879+0000 7f38298cf640 1 -- 172.21.15.98:0/3194321321 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38240ad240 con 0x7f38240b4700 2024-03-25T19:42:02.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.879+0000 7f38298cf640 1 librados: init done 2024-03-25T19:42:02.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.879+0000 7f38298cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:02.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.879+0000 7f38298cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:02.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.879+0000 7f38298cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:02.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:02.879+0000 7f38298cf640 1 -- 172.21.15.98:0/3194321321 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f382404f730 con 0x7f38240b4700 2024-03-25T19:42:02.911 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:02.911+0000 7f69b922b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:42:02.946 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:02.946+0000 7fbddaff1040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:42:02.990 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:02.991+0000 7fbddaff1040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:42:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.090+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3818001d00 con 0x7f38240b4700 2024-03-25T19:42:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.090+0000 7f380bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.090+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f38180037b0 con 0x7f38240b4700 2024-03-25T19:42:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.090+0000 7f380bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.090+0000 7f380bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.090+0000 7f380bfff640 10 monclient: dump: 2024-03-25T19:42:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:03.091 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.090+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3818003ca0 con 0x7f38240b4700 2024-03-25T19:42:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.090+0000 7f38217fa640 4 set_mon_vals no callback set 2024-03-25T19:42:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.092+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2714059186 0 0) 0x7f38180053a0 con 0x7f38240b4700 2024-03-25T19:42:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.092+0000 7f380bfff640 10 client.4127.objecter ms_dispatch 0x7f3824000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:42:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.092+0000 7f380bfff640 3 client.4127.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:42:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.092+0000 7f380bfff640 3 client.4127.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:42:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.092+0000 7f380bfff640 20 client.4127.objecter dump_active .. 0 homeless 2024-03-25T19:42:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.092+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3818004f80 con 0x7f38240b4700 2024-03-25T19:42:03.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.092+0000 7f380bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:03.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.092+0000 7f380bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:03.101 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:03.101+0000 7f69b922b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:42:03.102 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:03.103+0000 7fbddaff1040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:42:03.146 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:03.146+0000 7f69b922b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:42:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.165+0000 7f38298cf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ebc66313-ec95-428e-9fa9-559c14a21152", "id": 2}] 2024-03-25T19:42:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.165+0000 7f38298cf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ebc66313-ec95-428e-9fa9-559c14a21152", "id": 2}] 2024-03-25T19:42:03.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.165+0000 7f38298cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:03.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.165+0000 7f38298cf640 1 -- 172.21.15.98:0/3194321321 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd new", "uuid": "ebc66313-ec95-428e-9fa9-559c14a21152", "id": 2} v 0) v1 -- 0x7f38241d78b0 con 0x7f38240b4700 2024-03-25T19:42:03.200 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:03.200+0000 7fbddaff1040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:42:03.268 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:03.269+0000 7f69b922b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:42:03.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.295+0000 7f380bfff640 1 -- 172.21.15.98:0/3194321321 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ebc66313-ec95-428e-9fa9-559c14a21152", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 4064783295 0 216599125) 0x7f3818004180 con 0x7f38240b4700 2024-03-25T19:42:03.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.295+0000 7f380bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ebc66313-ec95-428e-9fa9-559c14a21152", "id": 2}] 2024-03-25T19:42:03.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.295+0000 7f380bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:03.295 INFO:teuthology.orchestra.run.smithi098.stdout:2 2024-03-25T19:42:03.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.296+0000 7f38298cf640 10 librados: watch_flush enter 2024-03-25T19:42:03.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.296+0000 7f38298cf640 10 librados: watch_flush exit 2024-03-25T19:42:03.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.296+0000 7f38298cf640 20 client.4127.objecter shutdown clearing up homeless session... 2024-03-25T19:42:03.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.296+0000 7f38298cf640 10 client.4127.objecter successfully canceled tick 2024-03-25T19:42:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.296+0000 7f38298cf640 10 monclient: shutdown 2024-03-25T19:42:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.296+0000 7f38298cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.296+0000 7f38298cf640 1 -- 172.21.15.98:0/3194321321 >> v1:172.21.15.81:6790/0 conn(0x7f38240b4700 legacy=0x7f38240acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.296+0000 7f38298cf640 1 -- 172.21.15.98:0/3194321321 shutdown_connections 2024-03-25T19:42:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.296+0000 7f38298cf640 1 -- 172.21.15.98:0/3194321321 >> 172.21.15.98:0/3194321321 conn(0x7f38240668e0 msgr2=0x7f38240b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.296+0000 7f38298cf640 1 -- 172.21.15.98:0/3194321321 shutdown_connections 2024-03-25T19:42:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.296+0000 7f38298cf640 1 -- 172.21.15.98:0/3194321321 wait complete. 2024-03-25T19:42:03.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.296+0000 7f38298cf640 1 librados: shutdown 2024-03-25T19:42:03.306 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new be8865f5-e87c-4abe-9bc2-c8e9aa7123bf 3 2024-03-25T19:42:03.361 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:03.362+0000 7fbddaff1040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:42:03.367 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:03.368+0000 7f69b922b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:42:03.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.405+0000 7f8f1ab96640 10 monclient: get_monmap_and_config 2024-03-25T19:42:03.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.405+0000 7f8f1ab96640 10 monclient: build_initial_monmap 2024-03-25T19:42:03.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.405+0000 7f8f1ab96640 10 monclient: monmap: 2024-03-25T19:42:03.405 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T19:42:03.405 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:03.406 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:42:03.406857+0000 2024-03-25T19:42:03.406 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:42:03.406857+0000 2024-03-25T19:42:03.406 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:03.406 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:03.406 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:03.406 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:42:03.406 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:42:03.406 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:03.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.406+0000 7f8f1ab96640 1 Processor -- start 2024-03-25T19:42:03.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.406+0000 7f8f1ab96640 1 -- start start 2024-03-25T19:42:03.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.406+0000 7f8f1ab96640 10 monclient: init 2024-03-25T19:42:03.407 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:03.406+0000 7fbddaff1040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:42:03.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.407+0000 7f8f1ab96640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:03.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.407+0000 7f8f1ab96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:03.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.407+0000 7f8f1ab96640 10 monclient(hunting): picked mon.noname-b con 0x7f8f140b4700 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:03.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.407+0000 7f8f1ab96640 10 monclient(hunting): picked mon.noname-a con 0x7f8f14142a20 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:03.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.407+0000 7f8f1ab96640 10 monclient(hunting): picked mon.noname-c con 0x7f8f140a96b0 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:03.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.407+0000 7f8f1ab96640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f140a9d60 con 0x7f8f14142a20 2024-03-25T19:42:03.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.407+0000 7f8f1ab96640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1413ff10 con 0x7f8f140a96b0 2024-03-25T19:42:03.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.407+0000 7f8f1ab96640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f140aa1f0 con 0x7f8f140b4700 2024-03-25T19:42:03.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.407+0000 7f8f1ab96640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:03.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.407+0000 7f8f1ab96640 10 monclient(hunting): authenticate will time out at 3958.691993s 2024-03-25T19:42:03.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.407+0000 7f8f19133640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8f140a96b0 0x7f8f1413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51794/0 (socket says 172.21.15.98:51794) 2024-03-25T19:42:03.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.407+0000 7f8f19133640 1 -- 172.21.15.98:0/221554836 learned_addr learned my addr 172.21.15.98:0/221554836 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T19:42:03.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 1 -- 172.21.15.98:0/221554836 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725606556 0 0) 0x7f8f1413ff10 con 0x7f8f140a96b0 2024-03-25T19:42:03.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:42:03.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 1 -- 172.21.15.98:0/221554836 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f04003520 con 0x7f8f140a96b0 2024-03-25T19:42:03.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 1 -- 172.21.15.98:0/221554836 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 923980200 0 0) 0x7f8f140aa1f0 con 0x7f8f140b4700 2024-03-25T19:42:03.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:42:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 1 -- 172.21.15.98:0/221554836 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1413ff10 con 0x7f8f140b4700 2024-03-25T19:42:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 1 -- 172.21.15.98:0/221554836 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563604775 0 0) 0x7f8f140a9d60 con 0x7f8f14142a20 2024-03-25T19:42:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:42:03.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.410+0000 7f8f137fe640 1 -- 172.21.15.98:0/221554836 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f140aa1f0 con 0x7f8f14142a20 2024-03-25T19:42:03.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.412+0000 7f8f137fe640 1 -- 172.21.15.98:0/221554836 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4080608886 0 0) 0x7f8f1413ff10 con 0x7f8f140b4700 2024-03-25T19:42:03.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.412+0000 7f8f137fe640 1 -- 172.21.15.98:0/221554836 >> v1:172.21.15.98:6789/0 conn(0x7f8f140a96b0 legacy=0x7f8f1413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.412+0000 7f8f137fe640 1 -- 172.21.15.98:0/221554836 >> v1:172.21.15.81:6789/0 conn(0x7f8f14142a20 legacy=0x7f8f1413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.412+0000 7f8f137fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:03.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.412+0000 7f8f137fe640 1 monclient: found mon.noname-b 2024-03-25T19:42:03.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.412+0000 7f8f137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:03.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.412+0000 7f8f137fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:42:03.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.412+0000 7f8f137fe640 1 -- 172.21.15.98:0/221554836 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f140b4ae0 con 0x7f8f140b4700 2024-03-25T19:42:03.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.412+0000 7f8f137fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:03.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.412+0000 7f8f137fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:03.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.412+0000 7f8f137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:03.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.412+0000 7f8f137fe640 1 -- 172.21.15.98:0/221554836 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8f08001670 con 0x7f8f140b4700 2024-03-25T19:42:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.412+0000 7f8f1ab96640 5 monclient: authenticate success, global_id 4130 2024-03-25T19:42:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.412+0000 7f8f1ab96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.412+0000 7f8f137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.413+0000 7f8f137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.413+0000 7f8f137fe640 10 monclient: dump: 2024-03-25T19:42:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:03.414 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:03.415 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:03.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.413+0000 7f8f1ab96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:03.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.415+0000 7f8f137fe640 1 -- 172.21.15.98:0/221554836 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f080017d0 con 0x7f8f140b4700 2024-03-25T19:42:03.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.415+0000 7f8f137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:03.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.415+0000 7f8f1ab96640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:03.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.415+0000 7f8f1ab96640 4 set_mon_vals no callback set 2024-03-25T19:42:03.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.415+0000 7f8f137fe640 1 -- 172.21.15.98:0/221554836 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8f08001be0 con 0x7f8f140b4700 2024-03-25T19:42:03.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.415+0000 7f8f137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.415+0000 7f8f137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:03.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.415+0000 7f8f137fe640 10 monclient: dump: 2024-03-25T19:42:03.419 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:03.419 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:03.419 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.419 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.419 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.419 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:03.419 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:03.419 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:03.419 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:03.419 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:03.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.416+0000 7f8f1ab96640 10 monclient: shutdown 2024-03-25T19:42:03.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.416+0000 7f8f1ab96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:03.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.416+0000 7f8f1ab96640 1 -- 172.21.15.98:0/221554836 >> v1:172.21.15.81:6790/0 conn(0x7f8f140b4700 legacy=0x7f8f14142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.416+0000 7f8f1ab96640 1 -- 172.21.15.98:0/221554836 shutdown_connections 2024-03-25T19:42:03.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.416+0000 7f8f1ab96640 1 -- 172.21.15.98:0/221554836 >> 172.21.15.98:0/221554836 conn(0x7f8f140668e0 msgr2=0x7f8f14066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:03.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.416+0000 7f8f1ab96640 1 -- 172.21.15.98:0/221554836 shutdown_connections 2024-03-25T19:42:03.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.416+0000 7f8f1ab96640 1 -- 172.21.15.98:0/221554836 wait complete. 2024-03-25T19:42:03.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.416+0000 7f8f1ab96640 10 monclient: build_initial_monmap 2024-03-25T19:42:03.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.416+0000 7f8f1ab96640 10 monclient: monmap: 2024-03-25T19:42:03.420 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T19:42:03.420 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:03.420 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-25T19:42:03.420 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-25T19:42:03.420 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:03.420 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-25T19:42:03.420 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:03.421 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.417+0000 7f8f1ab96640 1 librados: starting msgr at 2024-03-25T19:42:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.417+0000 7f8f1ab96640 1 librados: starting objecter 2024-03-25T19:42:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.417+0000 7f8f1ab96640 1 Processor -- start 2024-03-25T19:42:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.417+0000 7f8f1ab96640 1 -- start start 2024-03-25T19:42:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.417+0000 7f8f1ab96640 1 librados: setting wanted keys 2024-03-25T19:42:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.417+0000 7f8f1ab96640 1 librados: calling monclient init 2024-03-25T19:42:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.417+0000 7f8f1ab96640 10 monclient: init 2024-03-25T19:42:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.417+0000 7f8f1ab96640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.417+0000 7f8f1ab96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.417+0000 7f8f1ab96640 10 monclient(hunting): picked mon.noname-b con 0x7f8f140a96b0 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.417+0000 7f8f1ab96640 10 monclient(hunting): picked mon.noname-a con 0x7f8f140b4700 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.418+0000 7f8f1ab96640 10 monclient(hunting): picked mon.noname-c con 0x7f8f14142a20 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:03.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.418+0000 7f8f1ab96640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f14134970 con 0x7f8f140b4700 2024-03-25T19:42:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.418+0000 7f8f1ab96640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1412e940 con 0x7f8f140a96b0 2024-03-25T19:42:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.418+0000 7f8f1ab96640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f14136080 con 0x7f8f14142a20 2024-03-25T19:42:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.418+0000 7f8f1ab96640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.418+0000 7f8f1ab96640 10 monclient(hunting): authenticate will time out at 3958.702759s 2024-03-25T19:42:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.418+0000 7f8f18932640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8f140a96b0 0x7f8f141da960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51800/0 (socket says 172.21.15.98:51800) 2024-03-25T19:42:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.418+0000 7f8f18932640 1 -- 172.21.15.98:0/3567342375 learned_addr learned my addr 172.21.15.98:0/3567342375 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T19:42:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.418+0000 7f8f117fa640 10 client.?.objecter ms_handle_connect 0x7f8f140a96b0 2024-03-25T19:42:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.418+0000 7f8f117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.418+0000 7f8f117fa640 10 client.?.objecter ms_handle_connect 0x7f8f140b4700 2024-03-25T19:42:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.418+0000 7f8f117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.418+0000 7f8f117fa640 10 client.?.objecter ms_handle_connect 0x7f8f14142a20 2024-03-25T19:42:03.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.418+0000 7f8f117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:03.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.420+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568477795 0 0) 0x7f8f1412e940 con 0x7f8f140a96b0 2024-03-25T19:42:03.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.420+0000 7f8f117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.420+0000 7f8f117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.420+0000 7f8f117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.420+0000 7f8f117fa640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:42:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.420+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef8004eb0 con 0x7f8f140a96b0 2024-03-25T19:42:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.420+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056999386 0 0) 0x7f8f14134970 con 0x7f8f140b4700 2024-03-25T19:42:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.420+0000 7f8f117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.420+0000 7f8f117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.420+0000 7f8f117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.420+0000 7f8f117fa640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:42:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.420+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1412e940 con 0x7f8f140b4700 2024-03-25T19:42:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.421+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440520645 0 0) 0x7f8f14136080 con 0x7f8f14142a20 2024-03-25T19:42:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.421+0000 7f8f117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.421+0000 7f8f117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.421+0000 7f8f117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.421+0000 7f8f117fa640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:42:03.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.421+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f14134970 con 0x7f8f14142a20 2024-03-25T19:42:03.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.424+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3669576472 0 0) 0x7f8f14134970 con 0x7f8f14142a20 2024-03-25T19:42:03.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.424+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f14136080 con 0x7f8f14142a20 2024-03-25T19:42:03.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.424+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8f0c001cd0 con 0x7f8f14142a20 2024-03-25T19:42:03.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.424+0000 7f8f117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.424+0000 7f8f117fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:03.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.424+0000 7f8f117fa640 10 monclient(hunting): dump: 2024-03-25T19:42:03.426 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:03.426 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:03.426 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.426 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.427 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.427 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:03.427 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:03.427 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:03.427 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:03.427 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:03.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.424+0000 7f8f117fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:03.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.425+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4134368829 0 0) 0x7f8ef8004eb0 con 0x7f8f140a96b0 2024-03-25T19:42:03.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.425+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f14134970 con 0x7f8f140a96b0 2024-03-25T19:42:03.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.425+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8f08001be0 con 0x7f8f140a96b0 2024-03-25T19:42:03.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.425+0000 7f8f117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.425+0000 7f8f117fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:03.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.425+0000 7f8f117fa640 10 monclient(hunting): dump: 2024-03-25T19:42:03.427 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:03.428 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:03.428 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.428 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.428 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.428 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:03.428 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:03.428 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:03.428 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:03.428 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:03.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.426+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1674378401 0 0) 0x7f8f1412e940 con 0x7f8f140b4700 2024-03-25T19:42:03.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.426+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ef8004eb0 con 0x7f8f140b4700 2024-03-25T19:42:03.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.426+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8f000017d0 con 0x7f8f140b4700 2024-03-25T19:42:03.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.426+0000 7f8f117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.426+0000 7f8f117fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:03.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.426+0000 7f8f117fa640 10 monclient(hunting): dump: 2024-03-25T19:42:03.429 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:03.429 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:03.429 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.429 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.429 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.429 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:03.429 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:03.429 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:03.429 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:03.429 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:03.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2184893052 0 0) 0x7f8f14136080 con 0x7f8f14142a20 2024-03-25T19:42:03.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 >> v1:172.21.15.98:6789/0 conn(0x7f8f140a96b0 legacy=0x7f8f141da960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 >> v1:172.21.15.81:6789/0 conn(0x7f8f140b4700 legacy=0x7f8f1412db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f117fa640 10 monclient: _finish_hunting 0 2024-03-25T19:42:03.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f117fa640 1 monclient: found mon.c 2024-03-25T19:42:03.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:03.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f117fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:03.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f141db070 con 0x7f8f14142a20 2024-03-25T19:42:03.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f117fa640 10 monclient: _finish_auth 0 2024-03-25T19:42:03.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f117fa640 10 monclient: _check_auth_tickets 2024-03-25T19:42:03.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:03.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f1ab96640 5 monclient: authenticate success, global_id 4133 2024-03-25T19:42:03.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f1ab96640 10 monclient: _renew_subs 2024-03-25T19:42:03.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f1ab96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:03.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f1ab96640 1 -- 172.21.15.98:0/3567342375 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f14136780 con 0x7f8f14142a20 2024-03-25T19:42:03.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f1ab96640 10 client.4133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f1ab96640 10 monclient: _renew_subs 2024-03-25T19:42:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f1ab96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f1ab96640 1 -- 172.21.15.98:0/3567342375 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f14134550 con 0x7f8f14142a20 2024-03-25T19:42:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.428+0000 7f8f1ab96640 1 librados: init done 2024-03-25T19:42:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.429+0000 7f8f1ab96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.429+0000 7f8f1ab96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.429+0000 7f8f1ab96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:03.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.429+0000 7f8f1ab96640 1 -- 172.21.15.98:0/3567342375 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1404f730 con 0x7f8f14142a20 2024-03-25T19:42:03.533 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:03.533+0000 7f69b922b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:42:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.533+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f0c0020e0 con 0x7f8f14142a20 2024-03-25T19:42:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.533+0000 7f8f117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.533+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8f0c004d00 con 0x7f8f14142a20 2024-03-25T19:42:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.533+0000 7f8f117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.533+0000 7f8f117fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:03.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.533+0000 7f8f117fa640 10 monclient: dump: 2024-03-25T19:42:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:03.534 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.533+0000 7f8f12ffd640 4 set_mon_vals no callback set 2024-03-25T19:42:03.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.533+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8f0c0051f0 con 0x7f8f14142a20 2024-03-25T19:42:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.534+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3676218506 0 0) 0x7f8f0c006760 con 0x7f8f14142a20 2024-03-25T19:42:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.534+0000 7f8f117fa640 10 client.4133.objecter ms_dispatch 0x7f8f14000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:42:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.534+0000 7f8f117fa640 3 client.4133.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:42:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.534+0000 7f8f117fa640 3 client.4133.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:42:03.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.534+0000 7f8f117fa640 20 client.4133.objecter dump_active .. 0 homeless 2024-03-25T19:42:03.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.534+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8f0c0062d0 con 0x7f8f14142a20 2024-03-25T19:42:03.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.534+0000 7f8f117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:03.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.534+0000 7f8f117fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:03.567 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:03.568+0000 7fbddaff1040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:42:03.582 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:03.583+0000 7f69b922b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:42:03.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.607+0000 7f8f1ab96640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "be8865f5-e87c-4abe-9bc2-c8e9aa7123bf", "id": 3}] 2024-03-25T19:42:03.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.607+0000 7f8f1ab96640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "be8865f5-e87c-4abe-9bc2-c8e9aa7123bf", "id": 3}] 2024-03-25T19:42:03.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.607+0000 7f8f1ab96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:03.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.607+0000 7f8f1ab96640 1 -- 172.21.15.98:0/3567342375 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd new", "uuid": "be8865f5-e87c-4abe-9bc2-c8e9aa7123bf", "id": 3} v 0) v1 -- 0x7f8f1413e650 con 0x7f8f14142a20 2024-03-25T19:42:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.699+0000 7f8f117fa640 1 -- 172.21.15.98:0/3567342375 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "be8865f5-e87c-4abe-9bc2-c8e9aa7123bf", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3919723311 0 525046306) 0x7f8f0c006a50 con 0x7f8f14142a20 2024-03-25T19:42:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.699+0000 7f8f117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "be8865f5-e87c-4abe-9bc2-c8e9aa7123bf", "id": 3}] 2024-03-25T19:42:03.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.699+0000 7f8f117fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:03.699 INFO:teuthology.orchestra.run.smithi098.stdout:3 2024-03-25T19:42:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.700+0000 7f8f1ab96640 10 librados: watch_flush enter 2024-03-25T19:42:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.700+0000 7f8f1ab96640 10 librados: watch_flush exit 2024-03-25T19:42:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.700+0000 7f8f1ab96640 20 client.4133.objecter shutdown clearing up homeless session... 2024-03-25T19:42:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.700+0000 7f8f1ab96640 10 client.4133.objecter successfully canceled tick 2024-03-25T19:42:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.700+0000 7f8f1ab96640 10 monclient: shutdown 2024-03-25T19:42:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.700+0000 7f8f1ab96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.700+0000 7f8f1ab96640 1 -- 172.21.15.98:0/3567342375 >> v1:172.21.15.81:6790/0 conn(0x7f8f14142a20 legacy=0x7f8f1412e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.700+0000 7f8f1ab96640 1 -- 172.21.15.98:0/3567342375 shutdown_connections 2024-03-25T19:42:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.700+0000 7f8f1ab96640 1 -- 172.21.15.98:0/3567342375 >> 172.21.15.98:0/3567342375 conn(0x7f8f140668e0 msgr2=0x7f8f1413f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.700+0000 7f8f1ab96640 1 -- 172.21.15.98:0/3567342375 shutdown_connections 2024-03-25T19:42:03.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.700+0000 7f8f1ab96640 1 -- 172.21.15.98:0/3567342375 wait complete. 2024-03-25T19:42:03.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.700+0000 7f8f1ab96640 1 librados: shutdown 2024-03-25T19:42:03.710 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 35265d96-239a-479d-92fb-2508747e7475 4 2024-03-25T19:42:03.749 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:03.749+0000 7f69b922b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:42:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.810+0000 7f81e85ef640 10 monclient: get_monmap_and_config 2024-03-25T19:42:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.810+0000 7f81e85ef640 10 monclient: build_initial_monmap 2024-03-25T19:42:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.810+0000 7f81e85ef640 10 monclient: monmap: 2024-03-25T19:42:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T19:42:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:42:03.811570+0000 2024-03-25T19:42:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:42:03.811570+0000 2024-03-25T19:42:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:42:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:42:03.811 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.811+0000 7f81e85ef640 1 Processor -- start 2024-03-25T19:42:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.811+0000 7f81e85ef640 1 -- start start 2024-03-25T19:42:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.811+0000 7f81e85ef640 10 monclient: init 2024-03-25T19:42:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.811+0000 7f81e85ef640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.811+0000 7f81e85ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.812+0000 7f81e85ef640 10 monclient(hunting): picked mon.noname-a con 0x7f81e00b4700 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.812+0000 7f81e85ef640 10 monclient(hunting): picked mon.noname-c con 0x7f81e0142a20 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.812+0000 7f81e85ef640 10 monclient(hunting): picked mon.noname-b con 0x7f81e00a96b0 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.812+0000 7f81e85ef640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00a9d60 con 0x7f81e00b4700 2024-03-25T19:42:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.812+0000 7f81e85ef640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e013ff10 con 0x7f81e0142a20 2024-03-25T19:42:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.812+0000 7f81e85ef640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00aa1f0 con 0x7f81e00a96b0 2024-03-25T19:42:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.812+0000 7f81e85ef640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.812+0000 7f81e85ef640 10 monclient(hunting): authenticate will time out at 3959.096802s 2024-03-25T19:42:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.812+0000 7f81e5b8a640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f81e0142a20 0x7f81e013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51814/0 (socket says 172.21.15.98:51814) 2024-03-25T19:42:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.812+0000 7f81e5b8a640 1 -- 172.21.15.98:0/3760965562 learned_addr learned my addr 172.21.15.98:0/3760965562 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T19:42:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.815+0000 7f81e5389640 1 -- 172.21.15.98:0/3760965562 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076118334 0 0) 0x7f81e013ff10 con 0x7f81e0142a20 2024-03-25T19:42:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.815+0000 7f81e5389640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.815+0000 7f81e5389640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.815+0000 7f81e5389640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.815+0000 7f81e5389640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:42:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.815+0000 7f81e5389640 1 -- 172.21.15.98:0/3760965562 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc003520 con 0x7f81e0142a20 2024-03-25T19:42:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.815+0000 7f81e5389640 1 -- 172.21.15.98:0/3760965562 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 673201816 0 0) 0x7f81e00aa1f0 con 0x7f81e00a96b0 2024-03-25T19:42:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.815+0000 7f81e5389640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.815+0000 7f81e5389640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.815+0000 7f81e5389640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.815+0000 7f81e5389640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:42:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.815+0000 7f81e5389640 1 -- 172.21.15.98:0/3760965562 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e013ff10 con 0x7f81e00a96b0 2024-03-25T19:42:03.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.816+0000 7f81e5389640 1 -- 172.21.15.98:0/3760965562 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300052540 0 0) 0x7f81e00a9d60 con 0x7f81e00b4700 2024-03-25T19:42:03.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.816+0000 7f81e5389640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.816+0000 7f81e5389640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.816+0000 7f81e5389640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.816+0000 7f81e5389640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:42:03.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.816+0000 7f81e5389640 1 -- 172.21.15.98:0/3760965562 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e00aa1f0 con 0x7f81e00b4700 2024-03-25T19:42:03.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e5389640 1 -- 172.21.15.98:0/3760965562 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2539456733 0 0) 0x7f81cc003520 con 0x7f81e0142a20 2024-03-25T19:42:03.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e5389640 1 -- 172.21.15.98:0/3760965562 >> v1:172.21.15.81:6790/0 conn(0x7f81e00a96b0 legacy=0x7f81e013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e5389640 1 -- 172.21.15.98:0/3760965562 >> v1:172.21.15.81:6789/0 conn(0x7f81e00b4700 legacy=0x7f81e0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e5389640 10 monclient: _finish_hunting 0 2024-03-25T19:42:03.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e5389640 1 monclient: found mon.noname-c 2024-03-25T19:42:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e5389640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e5389640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:42:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e5389640 1 -- 172.21.15.98:0/3760965562 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e00b4ae0 con 0x7f81e0142a20 2024-03-25T19:42:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e5389640 10 monclient: _finish_auth 0 2024-03-25T19:42:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e5389640 10 monclient: _check_auth_tickets 2024-03-25T19:42:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e5389640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e5389640 1 -- 172.21.15.98:0/3760965562 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f81d0001690 con 0x7f81e0142a20 2024-03-25T19:42:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e85ef640 5 monclient: authenticate success, global_id 4141 2024-03-25T19:42:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e85ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e5389640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e5389640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e5389640 10 monclient: dump: 2024-03-25T19:42:03.819 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:03.820 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:03.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.818+0000 7f81e85ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:03.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.820+0000 7f81e5389640 1 -- 172.21.15.98:0/3760965562 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81d00017f0 con 0x7f81e0142a20 2024-03-25T19:42:03.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.820+0000 7f81e5389640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:03.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.820+0000 7f81e5389640 1 -- 172.21.15.98:0/3760965562 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f81d0001c00 con 0x7f81e0142a20 2024-03-25T19:42:03.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.820+0000 7f81e85ef640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:03.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.820+0000 7f81e85ef640 4 set_mon_vals no callback set 2024-03-25T19:42:03.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.820+0000 7f81e5389640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.820+0000 7f81e5389640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:03.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.820+0000 7f81e5389640 10 monclient: dump: 2024-03-25T19:42:03.822 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:03.822 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:03.822 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:03.823 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.821+0000 7f81e85ef640 10 monclient: shutdown 2024-03-25T19:42:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.821+0000 7f81e85ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.821+0000 7f81e85ef640 1 -- 172.21.15.98:0/3760965562 >> v1:172.21.15.98:6789/0 conn(0x7f81e0142a20 legacy=0x7f81e013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.821+0000 7f81e85ef640 1 -- 172.21.15.98:0/3760965562 shutdown_connections 2024-03-25T19:42:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.821+0000 7f81e85ef640 1 -- 172.21.15.98:0/3760965562 >> 172.21.15.98:0/3760965562 conn(0x7f81e00668e0 msgr2=0x7f81e0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.821+0000 7f81e85ef640 1 -- 172.21.15.98:0/3760965562 shutdown_connections 2024-03-25T19:42:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.821+0000 7f81e85ef640 1 -- 172.21.15.98:0/3760965562 wait complete. 2024-03-25T19:42:03.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.822+0000 7f81e85ef640 10 monclient: build_initial_monmap 2024-03-25T19:42:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.822+0000 7f81e85ef640 10 monclient: monmap: 2024-03-25T19:42:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T19:42:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-25T19:42:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-25T19:42:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-25T19:42:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:03.824 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.822+0000 7f81e85ef640 1 librados: starting msgr at 2024-03-25T19:42:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.822+0000 7f81e85ef640 1 librados: starting objecter 2024-03-25T19:42:03.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.822+0000 7f81e85ef640 1 Processor -- start 2024-03-25T19:42:03.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.822+0000 7f81e85ef640 1 -- start start 2024-03-25T19:42:03.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.822+0000 7f81e85ef640 1 librados: setting wanted keys 2024-03-25T19:42:03.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.822+0000 7f81e85ef640 1 librados: calling monclient init 2024-03-25T19:42:03.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.822+0000 7f81e85ef640 10 monclient: init 2024-03-25T19:42:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.823+0000 7f81e85ef640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.823+0000 7f81e85ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.823+0000 7f81e85ef640 10 monclient(hunting): picked mon.noname-a con 0x7f81e00a96b0 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.823+0000 7f81e85ef640 10 monclient(hunting): picked mon.noname-b con 0x7f81e00b4700 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.823+0000 7f81e85ef640 10 monclient(hunting): picked mon.noname-c con 0x7f81e0142a20 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.823+0000 7f81e85ef640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e0139ce0 con 0x7f81e00a96b0 2024-03-25T19:42:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.823+0000 7f81e85ef640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00ac620 con 0x7f81e00b4700 2024-03-25T19:42:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.823+0000 7f81e85ef640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00b1700 con 0x7f81e0142a20 2024-03-25T19:42:03.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.823+0000 7f81e85ef640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.823+0000 7f81e85ef640 10 monclient(hunting): authenticate will time out at 3959.108011s 2024-03-25T19:42:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.823+0000 7f81e5b8a640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f81e00b4700 0x7f81e013c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51826/0 (socket says 172.21.15.98:51826) 2024-03-25T19:42:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.823+0000 7f81e5b8a640 1 -- 172.21.15.98:0/2613515368 learned_addr learned my addr 172.21.15.98:0/2613515368 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T19:42:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.824+0000 7f81d6ffd640 10 client.?.objecter ms_handle_connect 0x7f81e00b4700 2024-03-25T19:42:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.824+0000 7f81d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.824+0000 7f81d6ffd640 10 client.?.objecter ms_handle_connect 0x7f81e00a96b0 2024-03-25T19:42:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.824+0000 7f81d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.825+0000 7f81d6ffd640 10 client.?.objecter ms_handle_connect 0x7f81e0142a20 2024-03-25T19:42:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.825+0000 7f81d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.825+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063469290 0 0) 0x7f81e00ac620 con 0x7f81e00b4700 2024-03-25T19:42:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.825+0000 7f81d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.825+0000 7f81d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.825+0000 7f81d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.825+0000 7f81d6ffd640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:42:03.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.825+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81c4004eb0 con 0x7f81e00b4700 2024-03-25T19:42:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.826+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731944879 0 0) 0x7f81e0139ce0 con 0x7f81e00a96b0 2024-03-25T19:42:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.826+0000 7f81d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.826+0000 7f81d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.826+0000 7f81d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.826+0000 7f81d6ffd640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:42:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.826+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e00ac620 con 0x7f81e00a96b0 2024-03-25T19:42:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.826+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300266572 0 0) 0x7f81e00b1700 con 0x7f81e0142a20 2024-03-25T19:42:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.826+0000 7f81d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.826+0000 7f81d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.826+0000 7f81d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.826+0000 7f81d6ffd640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:42:03.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.826+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e0139ce0 con 0x7f81e0142a20 2024-03-25T19:42:03.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.830+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3113677278 0 0) 0x7f81c4004eb0 con 0x7f81e00b4700 2024-03-25T19:42:03.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.830+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81e00b1700 con 0x7f81e00b4700 2024-03-25T19:42:03.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.830+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f81d0001bc0 con 0x7f81e00b4700 2024-03-25T19:42:03.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.830+0000 7f81d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.830+0000 7f81d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:03.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.830+0000 7f81d6ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:03.830 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:03.830 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:03.830 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.831 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.831 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.831 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:03.831 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:03.831 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:03.831 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:03.831 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:03.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.830+0000 7f81d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.831+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3552968592 0 0) 0x7f81e0139ce0 con 0x7f81e0142a20 2024-03-25T19:42:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.831+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4004eb0 con 0x7f81e0142a20 2024-03-25T19:42:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.831+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f81d8001cb0 con 0x7f81e0142a20 2024-03-25T19:42:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.831+0000 7f81d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.831+0000 7f81d6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.831+0000 7f81d6ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.832 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:03.833 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.831+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2097767275 0 0) 0x7f81e00ac620 con 0x7f81e00a96b0 2024-03-25T19:42:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.831+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81e0139ce0 con 0x7f81e00a96b0 2024-03-25T19:42:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.831+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f81dc001870 con 0x7f81e00a96b0 2024-03-25T19:42:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.831+0000 7f81d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.831+0000 7f81d6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.831+0000 7f81d6ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:03.833 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.834 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.834 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.834 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:03.834 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:03.834 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:03.834 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:03.834 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:03.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 364376765 0 0) 0x7f81e00b1700 con 0x7f81e00b4700 2024-03-25T19:42:03.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 >> v1:172.21.15.81:6790/0 conn(0x7f81e0142a20 legacy=0x7f81e01e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 >> v1:172.21.15.81:6789/0 conn(0x7f81e00a96b0 legacy=0x7f81e013b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:03.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:03.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81d6ffd640 1 monclient: found mon.b 2024-03-25T19:42:03.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:03.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81d6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:03.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e014e6d0 con 0x7f81e00b4700 2024-03-25T19:42:03.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81d6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:03.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:03.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:03.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81e85ef640 5 monclient: authenticate success, global_id 4144 2024-03-25T19:42:03.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81e85ef640 10 monclient: _renew_subs 2024-03-25T19:42:03.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81e85ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:03.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81e85ef640 1 -- 172.21.15.98:0/2613515368 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e013c900 con 0x7f81e00b4700 2024-03-25T19:42:03.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81e85ef640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:03.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81e85ef640 10 monclient: _renew_subs 2024-03-25T19:42:03.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.833+0000 7f81e85ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:03.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.834+0000 7f81e85ef640 1 -- 172.21.15.98:0/2613515368 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e013a350 con 0x7f81e00b4700 2024-03-25T19:42:03.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.834+0000 7f81e85ef640 1 librados: init done 2024-03-25T19:42:03.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.835+0000 7f81e85ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:03.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.835+0000 7f81e85ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:03.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.835+0000 7f81e85ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:03.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.835+0000 7f81e85ef640 1 -- 172.21.15.98:0/2613515368 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e014e920 con 0x7f81e00b4700 2024-03-25T19:42:03.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.930+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81d00011f0 con 0x7f81e00b4700 2024-03-25T19:42:03.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.930+0000 7f81d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:03.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.930+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f81d0004c30 con 0x7f81e00b4700 2024-03-25T19:42:03.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.930+0000 7f81d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:03.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.930+0000 7f81d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:03.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.930+0000 7f81d6ffd640 10 monclient: dump: 2024-03-25T19:42:03.931 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:03.931 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:03.931 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.931 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:03.931 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:03.932 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.930+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f81d0005160 con 0x7f81e00b4700 2024-03-25T19:42:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.930+0000 7f81e4b88640 4 set_mon_vals no callback set 2024-03-25T19:42:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.931+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 797723854 0 0) 0x7f81d0006850 con 0x7f81e00b4700 2024-03-25T19:42:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.931+0000 7f81d6ffd640 10 client.4144.objecter ms_dispatch 0x7f81e0000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:42:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.931+0000 7f81d6ffd640 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:42:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.931+0000 7f81d6ffd640 3 client.4144.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:42:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.931+0000 7f81d6ffd640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T19:42:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.931+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f81d0006bd0 con 0x7f81e00b4700 2024-03-25T19:42:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.931+0000 7f81d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:03.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:03.931+0000 7f81d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:04.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.001+0000 7f81e85ef640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "35265d96-239a-479d-92fb-2508747e7475", "id": 4}] 2024-03-25T19:42:04.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.001+0000 7f81e85ef640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "35265d96-239a-479d-92fb-2508747e7475", "id": 4}] 2024-03-25T19:42:04.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.001+0000 7f81e85ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:04.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.002+0000 7f81e85ef640 1 -- 172.21.15.98:0/2613515368 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd new", "uuid": "35265d96-239a-479d-92fb-2508747e7475", "id": 4} v 0) v1 -- 0x7f81e004f730 con 0x7f81e00b4700 2024-03-25T19:42:04.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.077+0000 7f81d6ffd640 1 -- 172.21.15.98:0/2613515368 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "35265d96-239a-479d-92fb-2508747e7475", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3138262798 0 1697012583) 0x7f81d0006470 con 0x7f81e00b4700 2024-03-25T19:42:04.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.078+0000 7f81d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "35265d96-239a-479d-92fb-2508747e7475", "id": 4}] 2024-03-25T19:42:04.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.078+0000 7f81d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:04.078 INFO:teuthology.orchestra.run.smithi098.stdout:4 2024-03-25T19:42:04.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.078+0000 7f81e85ef640 10 librados: watch_flush enter 2024-03-25T19:42:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.078+0000 7f81e85ef640 10 librados: watch_flush exit 2024-03-25T19:42:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.078+0000 7f81e85ef640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T19:42:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.078+0000 7f81e85ef640 10 client.4144.objecter successfully canceled tick 2024-03-25T19:42:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.078+0000 7f81e85ef640 10 monclient: shutdown 2024-03-25T19:42:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.078+0000 7f81e85ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.078+0000 7f81e85ef640 1 -- 172.21.15.98:0/2613515368 >> v1:172.21.15.98:6789/0 conn(0x7f81e00b4700 legacy=0x7f81e013c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.078+0000 7f81e85ef640 1 -- 172.21.15.98:0/2613515368 shutdown_connections 2024-03-25T19:42:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.078+0000 7f81e85ef640 1 -- 172.21.15.98:0/2613515368 >> 172.21.15.98:0/2613515368 conn(0x7f81e00668e0 msgr2=0x7f81e00b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.079+0000 7f81e85ef640 1 -- 172.21.15.98:0/2613515368 shutdown_connections 2024-03-25T19:42:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.079+0000 7f81e85ef640 1 -- 172.21.15.98:0/2613515368 wait complete. 2024-03-25T19:42:04.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.079+0000 7f81e85ef640 1 librados: shutdown 2024-03-25T19:42:04.088 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new bbbc4f31-6814-4509-a08a-d30bce723396 5 2024-03-25T19:42:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.190+0000 7f96f9c4a640 10 monclient: get_monmap_and_config 2024-03-25T19:42:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.190+0000 7f96f9c4a640 10 monclient: build_initial_monmap 2024-03-25T19:42:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.190+0000 7f96f9c4a640 10 monclient: monmap: 2024-03-25T19:42:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T19:42:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:42:04.191142+0000 2024-03-25T19:42:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:42:04.191142+0000 2024-03-25T19:42:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:04.190 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:04.191 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:42:04.191 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:42:04.191 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.190+0000 7f96f9c4a640 1 Processor -- start 2024-03-25T19:42:04.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.190+0000 7f96f9c4a640 1 -- start start 2024-03-25T19:42:04.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.190+0000 7f96f9c4a640 10 monclient: init 2024-03-25T19:42:04.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.191+0000 7f96f9c4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:04.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.191+0000 7f96f9c4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:04.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.191+0000 7f96f9c4a640 10 monclient(hunting): picked mon.noname-a con 0x7f96f40b4700 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:04.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.191+0000 7f96f9c4a640 10 monclient(hunting): picked mon.noname-c con 0x7f96f4142a20 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:04.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.191+0000 7f96f9c4a640 10 monclient(hunting): picked mon.noname-b con 0x7f96f40a96b0 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:04.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.191+0000 7f96f9c4a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f40a9d60 con 0x7f96f40b4700 2024-03-25T19:42:04.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.191+0000 7f96f9c4a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f413ff10 con 0x7f96f4142a20 2024-03-25T19:42:04.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.191+0000 7f96f9c4a640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f40aa1f0 con 0x7f96f40a96b0 2024-03-25T19:42:04.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.191+0000 7f96f9c4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:04.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.191+0000 7f96f9c4a640 10 monclient(hunting): authenticate will time out at 3959.476453s 2024-03-25T19:42:04.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.192+0000 7f96f2ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f96f4142a20 0x7f96f413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51842/0 (socket says 172.21.15.98:51842) 2024-03-25T19:42:04.193 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.192+0000 7f96f2ffd640 1 -- 172.21.15.98:0/2822061393 learned_addr learned my addr 172.21.15.98:0/2822061393 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T19:42:04.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.194+0000 7f96f27fc640 1 -- 172.21.15.98:0/2822061393 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317780336 0 0) 0x7f96f413ff10 con 0x7f96f4142a20 2024-03-25T19:42:04.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.194+0000 7f96f27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.194+0000 7f96f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.194+0000 7f96f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.194+0000 7f96f27fc640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:42:04.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.194+0000 7f96f27fc640 1 -- 172.21.15.98:0/2822061393 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96dc003520 con 0x7f96f4142a20 2024-03-25T19:42:04.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.194+0000 7f96f27fc640 1 -- 172.21.15.98:0/2822061393 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820455243 0 0) 0x7f96f40aa1f0 con 0x7f96f40a96b0 2024-03-25T19:42:04.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.194+0000 7f96f27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.194+0000 7f96f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.194+0000 7f96f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.194+0000 7f96f27fc640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:42:04.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.194+0000 7f96f27fc640 1 -- 172.21.15.98:0/2822061393 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f413ff10 con 0x7f96f40a96b0 2024-03-25T19:42:04.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.195+0000 7f96f27fc640 1 -- 172.21.15.98:0/2822061393 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256567034 0 0) 0x7f96f40a9d60 con 0x7f96f40b4700 2024-03-25T19:42:04.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.195+0000 7f96f27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.195+0000 7f96f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.195+0000 7f96f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.195+0000 7f96f27fc640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:42:04.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.195+0000 7f96f27fc640 1 -- 172.21.15.98:0/2822061393 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f40aa1f0 con 0x7f96f40b4700 2024-03-25T19:42:04.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f27fc640 1 -- 172.21.15.98:0/2822061393 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1709665222 0 0) 0x7f96dc003520 con 0x7f96f4142a20 2024-03-25T19:42:04.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f27fc640 1 -- 172.21.15.98:0/2822061393 >> v1:172.21.15.81:6790/0 conn(0x7f96f40a96b0 legacy=0x7f96f413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f27fc640 1 -- 172.21.15.98:0/2822061393 >> v1:172.21.15.81:6789/0 conn(0x7f96f40b4700 legacy=0x7f96f4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:04.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f27fc640 1 monclient: found mon.noname-c 2024-03-25T19:42:04.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:04.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:42:04.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f27fc640 1 -- 172.21.15.98:0/2822061393 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f40b4ae0 con 0x7f96f4142a20 2024-03-25T19:42:04.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f27fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:04.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:04.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:04.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f27fc640 1 -- 172.21.15.98:0/2822061393 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f96e8001690 con 0x7f96f4142a20 2024-03-25T19:42:04.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f9c4a640 5 monclient: authenticate success, global_id 4147 2024-03-25T19:42:04.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f9c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:04.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:04.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f27fc640 10 monclient: dump: 2024-03-25T19:42:04.198 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:04.199 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:04.199 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.199 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.199 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.199 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:04.199 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:04.199 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:04.199 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:04.199 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.197+0000 7f96f9c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:04.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.199+0000 7f96f27fc640 1 -- 172.21.15.98:0/2822061393 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96e80017f0 con 0x7f96f4142a20 2024-03-25T19:42:04.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.199+0000 7f96f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:04.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.199+0000 7f96f27fc640 1 -- 172.21.15.98:0/2822061393 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f96e8001c40 con 0x7f96f4142a20 2024-03-25T19:42:04.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.199+0000 7f96f9c4a640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:04.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.199+0000 7f96f9c4a640 4 set_mon_vals no callback set 2024-03-25T19:42:04.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.199+0000 7f96f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.199+0000 7f96f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:04.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.199+0000 7f96f27fc640 10 monclient: dump: 2024-03-25T19:42:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:04.203 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.200+0000 7f96f9c4a640 10 monclient: shutdown 2024-03-25T19:42:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.200+0000 7f96f9c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.200+0000 7f96f9c4a640 1 -- 172.21.15.98:0/2822061393 >> v1:172.21.15.98:6789/0 conn(0x7f96f4142a20 legacy=0x7f96f413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.200+0000 7f96f9c4a640 1 -- 172.21.15.98:0/2822061393 shutdown_connections 2024-03-25T19:42:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.200+0000 7f96f9c4a640 1 -- 172.21.15.98:0/2822061393 >> 172.21.15.98:0/2822061393 conn(0x7f96f40668e0 msgr2=0x7f96f4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.200+0000 7f96f9c4a640 1 -- 172.21.15.98:0/2822061393 shutdown_connections 2024-03-25T19:42:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.200+0000 7f96f9c4a640 1 -- 172.21.15.98:0/2822061393 wait complete. 2024-03-25T19:42:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.200+0000 7f96f9c4a640 10 monclient: build_initial_monmap 2024-03-25T19:42:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.200+0000 7f96f9c4a640 10 monclient: monmap: 2024-03-25T19:42:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T19:42:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-25T19:42:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-25T19:42:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-25T19:42:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:04.204 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:04.205 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.201+0000 7f96f9c4a640 1 librados: starting msgr at 2024-03-25T19:42:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.201+0000 7f96f9c4a640 1 librados: starting objecter 2024-03-25T19:42:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.201+0000 7f96f9c4a640 1 Processor -- start 2024-03-25T19:42:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.201+0000 7f96f9c4a640 1 -- start start 2024-03-25T19:42:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.201+0000 7f96f9c4a640 1 librados: setting wanted keys 2024-03-25T19:42:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.201+0000 7f96f9c4a640 1 librados: calling monclient init 2024-03-25T19:42:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.201+0000 7f96f9c4a640 10 monclient: init 2024-03-25T19:42:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.202+0000 7f96f9c4a640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.202+0000 7f96f9c4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.202+0000 7f96f9c4a640 10 monclient(hunting): picked mon.noname-c con 0x7f96f40a96b0 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.202+0000 7f96f9c4a640 10 monclient(hunting): picked mon.noname-a con 0x7f96f40b4700 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.202+0000 7f96f9c4a640 10 monclient(hunting): picked mon.noname-b con 0x7f96f4142a20 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:04.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.202+0000 7f96f9c4a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f4139660 con 0x7f96f40b4700 2024-03-25T19:42:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.202+0000 7f96f9c4a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f4139140 con 0x7f96f4142a20 2024-03-25T19:42:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.202+0000 7f96f9c4a640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f41392d0 con 0x7f96f40a96b0 2024-03-25T19:42:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.202+0000 7f96f9c4a640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.202+0000 7f96f9c4a640 10 monclient(hunting): authenticate will time out at 3959.486924s 2024-03-25T19:42:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.202+0000 7f96f3fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f96f4142a20 0x7f96f41d7e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51856/0 (socket says 172.21.15.98:51856) 2024-03-25T19:42:04.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.202+0000 7f96f3fff640 1 -- 172.21.15.98:0/1718568335 learned_addr learned my addr 172.21.15.98:0/1718568335 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T19:42:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.203+0000 7f96dbfff640 10 client.?.objecter ms_handle_connect 0x7f96f4142a20 2024-03-25T19:42:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.203+0000 7f96dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:04.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.203+0000 7f96dbfff640 10 client.?.objecter ms_handle_connect 0x7f96f40b4700 2024-03-25T19:42:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.203+0000 7f96dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.204+0000 7f96dbfff640 10 client.?.objecter ms_handle_connect 0x7f96f40a96b0 2024-03-25T19:42:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.204+0000 7f96dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.206+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548763953 0 0) 0x7f96f4139140 con 0x7f96f4142a20 2024-03-25T19:42:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.206+0000 7f96dbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.206+0000 7f96dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.206+0000 7f96dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.206+0000 7f96dbfff640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:42:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.206+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96e0004eb0 con 0x7f96f4142a20 2024-03-25T19:42:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.206+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531758149 0 0) 0x7f96f41392d0 con 0x7f96f40a96b0 2024-03-25T19:42:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.206+0000 7f96dbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.206+0000 7f96dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.206+0000 7f96dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.206+0000 7f96dbfff640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:42:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.206+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f4139140 con 0x7f96f40a96b0 2024-03-25T19:42:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.207+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292717933 0 0) 0x7f96f4139660 con 0x7f96f40b4700 2024-03-25T19:42:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.207+0000 7f96dbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.207+0000 7f96dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.207+0000 7f96dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.207+0000 7f96dbfff640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:42:04.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.207+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96f41392d0 con 0x7f96f40b4700 2024-03-25T19:42:04.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.211+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3172847839 0 0) 0x7f96e0004eb0 con 0x7f96f4142a20 2024-03-25T19:42:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.211+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96f4139660 con 0x7f96f4142a20 2024-03-25T19:42:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.211+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f96ec001dd0 con 0x7f96f4142a20 2024-03-25T19:42:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.211+0000 7f96dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.211+0000 7f96dbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.211+0000 7f96dbfff640 10 monclient(hunting): dump: 2024-03-25T19:42:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:04.212 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.211+0000 7f96dbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.212+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1762798265 0 0) 0x7f96f4139140 con 0x7f96f40a96b0 2024-03-25T19:42:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.212+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96e0004eb0 con 0x7f96f40a96b0 2024-03-25T19:42:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.212+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f96e4001870 con 0x7f96f40a96b0 2024-03-25T19:42:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.212+0000 7f96dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.212+0000 7f96dbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.212+0000 7f96dbfff640 10 monclient(hunting): dump: 2024-03-25T19:42:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:04.214 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1342537065 0 0) 0x7f96f41392d0 con 0x7f96f40b4700 2024-03-25T19:42:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96f4139140 con 0x7f96f40b4700 2024-03-25T19:42:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f96e8001b00 con 0x7f96f40b4700 2024-03-25T19:42:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96dbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96dbfff640 10 monclient(hunting): dump: 2024-03-25T19:42:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:04.216 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3838543829 0 0) 0x7f96f4139660 con 0x7f96f4142a20 2024-03-25T19:42:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 >> v1:172.21.15.81:6790/0 conn(0x7f96f40a96b0 legacy=0x7f96f4138a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 >> v1:172.21.15.81:6789/0 conn(0x7f96f40b4700 legacy=0x7f96f41d7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96dbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96dbfff640 1 monclient: found mon.b 2024-03-25T19:42:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96dbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f4135970 con 0x7f96f4142a20 2024-03-25T19:42:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96dbfff640 10 monclient: _finish_auth 0 2024-03-25T19:42:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96dbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96f9c4a640 5 monclient: authenticate success, global_id 4150 2024-03-25T19:42:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96f9c4a640 10 monclient: _renew_subs 2024-03-25T19:42:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96f9c4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96f9c4a640 1 -- 172.21.15.98:0/1718568335 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f4135c20 con 0x7f96f4142a20 2024-03-25T19:42:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96f9c4a640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96f9c4a640 10 monclient: _renew_subs 2024-03-25T19:42:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96f9c4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96f9c4a640 1 -- 172.21.15.98:0/1718568335 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f41e3100 con 0x7f96f4142a20 2024-03-25T19:42:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.214+0000 7f96f9c4a640 1 librados: init done 2024-03-25T19:42:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.215+0000 7f96f9c4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.215+0000 7f96f9c4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.215+0000 7f96f9c4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:04.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.215+0000 7f96f9c4a640 1 -- 172.21.15.98:0/1718568335 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f404f730 con 0x7f96f4142a20 2024-03-25T19:42:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.337+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96ec003970 con 0x7f96f4142a20 2024-03-25T19:42:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.337+0000 7f96dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.337+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f96ec003d00 con 0x7f96f4142a20 2024-03-25T19:42:04.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.337+0000 7f96dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.337+0000 7f96dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:04.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.337+0000 7f96dbfff640 10 monclient: dump: 2024-03-25T19:42:04.338 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:04.338 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:04.338 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.338 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.338 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.338 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:04.338 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:04.338 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:04.338 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:04.338 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.337+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f96ec0042b0 con 0x7f96f4142a20 2024-03-25T19:42:04.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.337+0000 7f96f1ffb640 4 set_mon_vals no callback set 2024-03-25T19:42:04.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.337+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1046514922 0 0) 0x7f96ec006a90 con 0x7f96f4142a20 2024-03-25T19:42:04.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.337+0000 7f96dbfff640 10 client.4150.objecter ms_dispatch 0x7f96f4000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:42:04.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.337+0000 7f96dbfff640 3 client.4150.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:42:04.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.337+0000 7f96dbfff640 3 client.4150.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:42:04.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.337+0000 7f96dbfff640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-25T19:42:04.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.337+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f96ec018de0 con 0x7f96f4142a20 2024-03-25T19:42:04.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.337+0000 7f96dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:04.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.337+0000 7f96dbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:04.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.428+0000 7f96f9c4a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bbbc4f31-6814-4509-a08a-d30bce723396", "id": 5}] 2024-03-25T19:42:04.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.428+0000 7f96f9c4a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bbbc4f31-6814-4509-a08a-d30bce723396", "id": 5}] 2024-03-25T19:42:04.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.428+0000 7f96f9c4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:04.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.428+0000 7f96f9c4a640 1 -- 172.21.15.98:0/1718568335 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bbbc4f31-6814-4509-a08a-d30bce723396", "id": 5} v 0) v1 -- 0x7f96f40b2510 con 0x7f96f4142a20 2024-03-25T19:42:04.464 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:04.464+0000 7fbddaff1040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:42:04.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.502+0000 7f96dbfff640 1 -- 172.21.15.98:0/1718568335 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bbbc4f31-6814-4509-a08a-d30bce723396", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1020864090 0 1988412176) 0x7f96ec004630 con 0x7f96f4142a20 2024-03-25T19:42:04.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.502+0000 7f96dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bbbc4f31-6814-4509-a08a-d30bce723396", "id": 5}] 2024-03-25T19:42:04.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.502+0000 7f96dbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:04.502 INFO:teuthology.orchestra.run.smithi098.stdout:5 2024-03-25T19:42:04.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.502+0000 7f96f9c4a640 10 librados: watch_flush enter 2024-03-25T19:42:04.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.502+0000 7f96f9c4a640 10 librados: watch_flush exit 2024-03-25T19:42:04.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.502+0000 7f96f9c4a640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-25T19:42:04.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.502+0000 7f96f9c4a640 10 client.4150.objecter successfully canceled tick 2024-03-25T19:42:04.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.502+0000 7f96f9c4a640 10 monclient: shutdown 2024-03-25T19:42:04.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.502+0000 7f96f9c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:04.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.502+0000 7f96f9c4a640 1 -- 172.21.15.98:0/1718568335 >> v1:172.21.15.98:6789/0 conn(0x7f96f4142a20 legacy=0x7f96f41d7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.502+0000 7f96f9c4a640 1 -- 172.21.15.98:0/1718568335 shutdown_connections 2024-03-25T19:42:04.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.503+0000 7f96f9c4a640 1 -- 172.21.15.98:0/1718568335 >> 172.21.15.98:0/1718568335 conn(0x7f96f40668e0 msgr2=0x7f96f413f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:04.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.503+0000 7f96f9c4a640 1 -- 172.21.15.98:0/1718568335 shutdown_connections 2024-03-25T19:42:04.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.503+0000 7f96f9c4a640 1 -- 172.21.15.98:0/1718568335 wait complete. 2024-03-25T19:42:04.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.503+0000 7f96f9c4a640 1 librados: shutdown 2024-03-25T19:42:04.513 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new 36f755b4-5184-4bd5-8211-e880c2ae2551 6 2024-03-25T19:42:04.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.613+0000 7f166f6cf640 10 monclient: get_monmap_and_config 2024-03-25T19:42:04.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.613+0000 7f166f6cf640 10 monclient: build_initial_monmap 2024-03-25T19:42:04.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.614+0000 7f166f6cf640 10 monclient: monmap: 2024-03-25T19:42:04.614 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T19:42:04.614 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:04.614 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:42:04.614925+0000 2024-03-25T19:42:04.615 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:42:04.614925+0000 2024-03-25T19:42:04.615 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:04.615 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:04.615 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:04.615 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:42:04.615 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:42:04.615 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.614+0000 7f166f6cf640 1 Processor -- start 2024-03-25T19:42:04.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.614+0000 7f166f6cf640 1 -- start start 2024-03-25T19:42:04.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.614+0000 7f166f6cf640 10 monclient: init 2024-03-25T19:42:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.615+0000 7f166f6cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.615+0000 7f166f6cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.615+0000 7f166f6cf640 10 monclient(hunting): picked mon.noname-c con 0x7f16680b4700 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.615+0000 7f166f6cf640 10 monclient(hunting): picked mon.noname-a con 0x7f1668142a20 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.615+0000 7f166f6cf640 10 monclient(hunting): picked mon.noname-b con 0x7f16680a96b0 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.615+0000 7f166f6cf640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16680a9d60 con 0x7f1668142a20 2024-03-25T19:42:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.615+0000 7f166f6cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166813ff10 con 0x7f16680b4700 2024-03-25T19:42:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.615+0000 7f166f6cf640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16680aa1f0 con 0x7f16680a96b0 2024-03-25T19:42:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.615+0000 7f166f6cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.615+0000 7f166f6cf640 10 monclient(hunting): authenticate will time out at 3959.900077s 2024-03-25T19:42:04.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.615+0000 7f166d46b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f16680b4700 0x7f1668142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51862/0 (socket says 172.21.15.98:51862) 2024-03-25T19:42:04.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.615+0000 7f166d46b640 1 -- 172.21.15.98:0/2882240167 learned_addr learned my addr 172.21.15.98:0/2882240167 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T19:42:04.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.618+0000 7f165ffff640 1 -- 172.21.15.98:0/2882240167 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152750469 0 0) 0x7f166813ff10 con 0x7f16680b4700 2024-03-25T19:42:04.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.618+0000 7f165ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.618+0000 7f165ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.618+0000 7f165ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.618+0000 7f165ffff640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:42:04.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.618+0000 7f165ffff640 1 -- 172.21.15.98:0/2882240167 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1654003520 con 0x7f16680b4700 2024-03-25T19:42:04.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.618+0000 7f165ffff640 1 -- 172.21.15.98:0/2882240167 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036214987 0 0) 0x7f16680aa1f0 con 0x7f16680a96b0 2024-03-25T19:42:04.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.618+0000 7f165ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.618+0000 7f165ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.618+0000 7f165ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.618+0000 7f165ffff640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:42:04.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.618+0000 7f165ffff640 1 -- 172.21.15.98:0/2882240167 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f166813ff10 con 0x7f16680a96b0 2024-03-25T19:42:04.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.619+0000 7f165ffff640 1 -- 172.21.15.98:0/2882240167 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063159564 0 0) 0x7f16680a9d60 con 0x7f1668142a20 2024-03-25T19:42:04.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.619+0000 7f165ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.619+0000 7f165ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.619+0000 7f165ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.619+0000 7f165ffff640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:42:04.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.619+0000 7f165ffff640 1 -- 172.21.15.98:0/2882240167 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16680aa1f0 con 0x7f1668142a20 2024-03-25T19:42:04.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.621+0000 7f165ffff640 1 -- 172.21.15.98:0/2882240167 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 677751045 0 0) 0x7f1654003520 con 0x7f16680b4700 2024-03-25T19:42:04.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.621+0000 7f165ffff640 1 -- 172.21.15.98:0/2882240167 >> v1:172.21.15.81:6790/0 conn(0x7f16680a96b0 legacy=0x7f166813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.621+0000 7f165ffff640 1 -- 172.21.15.98:0/2882240167 >> v1:172.21.15.81:6789/0 conn(0x7f1668142a20 legacy=0x7f166813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.621+0000 7f165ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:04.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.621+0000 7f165ffff640 1 monclient: found mon.noname-c 2024-03-25T19:42:04.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.621+0000 7f165ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:04.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.621+0000 7f165ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:42:04.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.621+0000 7f165ffff640 1 -- 172.21.15.98:0/2882240167 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16680b4ae0 con 0x7f16680b4700 2024-03-25T19:42:04.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.621+0000 7f165ffff640 10 monclient: _finish_auth 0 2024-03-25T19:42:04.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.621+0000 7f165ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:04.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.621+0000 7f165ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:04.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.621+0000 7f166f6cf640 5 monclient: authenticate success, global_id 4153 2024-03-25T19:42:04.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.621+0000 7f165ffff640 1 -- 172.21.15.98:0/2882240167 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1660001690 con 0x7f16680b4700 2024-03-25T19:42:04.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.621+0000 7f165ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.621+0000 7f165ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:04.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.621+0000 7f165ffff640 10 monclient: dump: 2024-03-25T19:42:04.622 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:04.622 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:04.622 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.622 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.622 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.622 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:04.622 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:04.622 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:04.622 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:04.622 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.621+0000 7f166f6cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:04.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.622+0000 7f165ffff640 1 -- 172.21.15.98:0/2882240167 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16600017f0 con 0x7f16680b4700 2024-03-25T19:42:04.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.622+0000 7f165ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:04.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.622+0000 7f166f6cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:04.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.622+0000 7f166f6cf640 4 set_mon_vals no callback set 2024-03-25T19:42:04.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.622+0000 7f165ffff640 1 -- 172.21.15.98:0/2882240167 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1660001c00 con 0x7f16680b4700 2024-03-25T19:42:04.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.622+0000 7f165ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.622+0000 7f165ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:04.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.622+0000 7f165ffff640 10 monclient: dump: 2024-03-25T19:42:04.625 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:04.626 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:04.626 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.626 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.626 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.626 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:04.626 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:04.626 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:04.626 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:04.626 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.623+0000 7f166f6cf640 10 monclient: shutdown 2024-03-25T19:42:04.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.623+0000 7f166f6cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:04.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.623+0000 7f166f6cf640 1 -- 172.21.15.98:0/2882240167 >> v1:172.21.15.98:6789/0 conn(0x7f16680b4700 legacy=0x7f1668142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.623+0000 7f166f6cf640 1 -- 172.21.15.98:0/2882240167 shutdown_connections 2024-03-25T19:42:04.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.623+0000 7f166f6cf640 1 -- 172.21.15.98:0/2882240167 >> 172.21.15.98:0/2882240167 conn(0x7f16680668e0 msgr2=0x7f1668066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:04.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.623+0000 7f166f6cf640 1 -- 172.21.15.98:0/2882240167 shutdown_connections 2024-03-25T19:42:04.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.623+0000 7f166f6cf640 1 -- 172.21.15.98:0/2882240167 wait complete. 2024-03-25T19:42:04.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.623+0000 7f166f6cf640 10 monclient: build_initial_monmap 2024-03-25T19:42:04.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.623+0000 7f166f6cf640 10 monclient: monmap: 2024-03-25T19:42:04.627 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T19:42:04.627 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:04.627 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-25T19:42:04.627 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-25T19:42:04.627 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:04.627 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-25T19:42:04.627 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:04.627 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:04.627 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:04.627 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.624+0000 7f166f6cf640 1 librados: starting msgr at 2024-03-25T19:42:04.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.624+0000 7f166f6cf640 1 librados: starting objecter 2024-03-25T19:42:04.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.624+0000 7f166f6cf640 1 Processor -- start 2024-03-25T19:42:04.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.624+0000 7f166f6cf640 1 -- start start 2024-03-25T19:42:04.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.624+0000 7f166f6cf640 1 librados: setting wanted keys 2024-03-25T19:42:04.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.624+0000 7f166f6cf640 1 librados: calling monclient init 2024-03-25T19:42:04.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.624+0000 7f166f6cf640 10 monclient: init 2024-03-25T19:42:04.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.624+0000 7f166f6cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:04.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.624+0000 7f166f6cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:04.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.624+0000 7f166f6cf640 10 monclient(hunting): picked mon.noname-a con 0x7f16680a96b0 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:04.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.624+0000 7f166f6cf640 10 monclient(hunting): picked mon.noname-b con 0x7f16680b4700 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:04.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.624+0000 7f166f6cf640 10 monclient(hunting): picked mon.noname-c con 0x7f1668142a20 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:04.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.624+0000 7f166f6cf640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166813a630 con 0x7f16680a96b0 2024-03-25T19:42:04.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.624+0000 7f166f6cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166813a8a0 con 0x7f16680b4700 2024-03-25T19:42:04.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.624+0000 7f166f6cf640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166813ab80 con 0x7f1668142a20 2024-03-25T19:42:04.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.624+0000 7f166f6cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:04.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.624+0000 7f166f6cf640 10 monclient(hunting): authenticate will time out at 3959.909561s 2024-03-25T19:42:04.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.625+0000 7f166cc6a640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f16680b4700 0x7f1668130f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51866/0 (socket says 172.21.15.98:51866) 2024-03-25T19:42:04.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.625+0000 7f166cc6a640 1 -- 172.21.15.98:0/3212355169 learned_addr learned my addr 172.21.15.98:0/3212355169 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T19:42:04.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.625+0000 7f165dffb640 10 client.?.objecter ms_handle_connect 0x7f16680b4700 2024-03-25T19:42:04.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.625+0000 7f165dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:04.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.625+0000 7f165dffb640 10 client.?.objecter ms_handle_connect 0x7f1668142a20 2024-03-25T19:42:04.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.625+0000 7f165dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:04.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.625+0000 7f165dffb640 10 client.?.objecter ms_handle_connect 0x7f16680a96b0 2024-03-25T19:42:04.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.625+0000 7f165dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:04.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.627+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773303703 0 0) 0x7f166813a8a0 con 0x7f16680b4700 2024-03-25T19:42:04.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.627+0000 7f165dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.627+0000 7f165dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.627+0000 7f165dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.627+0000 7f165dffb640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:42:04.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.627+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f164c004eb0 con 0x7f16680b4700 2024-03-25T19:42:04.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.627+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558651343 0 0) 0x7f166813ab80 con 0x7f1668142a20 2024-03-25T19:42:04.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.627+0000 7f165dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.627+0000 7f165dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.627+0000 7f165dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.627+0000 7f165dffb640 10 monclient(hunting): my global_id is 4151 2024-03-25T19:42:04.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.627+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f166813a8a0 con 0x7f1668142a20 2024-03-25T19:42:04.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.628+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251659447 0 0) 0x7f166813a630 con 0x7f16680a96b0 2024-03-25T19:42:04.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.628+0000 7f165dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.628+0000 7f165dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.628+0000 7f165dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.628+0000 7f165dffb640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:42:04.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.628+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f166813ab80 con 0x7f16680a96b0 2024-03-25T19:42:04.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.631+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 825922657 0 0) 0x7f164c004eb0 con 0x7f16680b4700 2024-03-25T19:42:04.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.631+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f166813a630 con 0x7f16680b4700 2024-03-25T19:42:04.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.631+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1658001970 con 0x7f16680b4700 2024-03-25T19:42:04.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.631+0000 7f165dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.631+0000 7f165dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:04.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.631+0000 7f165dffb640 10 monclient(hunting): dump: 2024-03-25T19:42:04.632 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:04.633 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:04.633 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.633 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.633 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.633 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:04.633 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:04.633 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:04.633 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:04.633 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.632+0000 7f165dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:04.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.633+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3190424352 0 0) 0x7f166813a8a0 con 0x7f1668142a20 2024-03-25T19:42:04.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.633+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f164c004eb0 con 0x7f1668142a20 2024-03-25T19:42:04.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.633+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1664001cb0 con 0x7f1668142a20 2024-03-25T19:42:04.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.633+0000 7f165dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.633+0000 7f165dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:04.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.633+0000 7f165dffb640 10 monclient(hunting): dump: 2024-03-25T19:42:04.634 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:04.634 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:04.634 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.634 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.634 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.634 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:04.634 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:04.634 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:04.634 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:04.634 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.633+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 181949550 0 0) 0x7f166813ab80 con 0x7f16680a96b0 2024-03-25T19:42:04.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.633+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f166813a8a0 con 0x7f16680a96b0 2024-03-25T19:42:04.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.633+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1660001b00 con 0x7f16680a96b0 2024-03-25T19:42:04.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.633+0000 7f165dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.633+0000 7f165dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:04.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.633+0000 7f165dffb640 10 monclient(hunting): dump: 2024-03-25T19:42:04.636 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:04.636 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:04.636 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.636 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.636 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.636 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:04.636 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:04.637 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:04.637 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:04.637 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2452199895 0 0) 0x7f166813a630 con 0x7f16680b4700 2024-03-25T19:42:04.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 >> v1:172.21.15.81:6790/0 conn(0x7f1668142a20 legacy=0x7f16681e33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 >> v1:172.21.15.81:6789/0 conn(0x7f16680a96b0 legacy=0x7f16681392a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f165dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:04.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f165dffb640 1 monclient: found mon.b 2024-03-25T19:42:04.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f165dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:04.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f165dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:04.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1668138dc0 con 0x7f16680b4700 2024-03-25T19:42:04.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f165dffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:04.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f165dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:04.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f165dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:04.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f166f6cf640 5 monclient: authenticate success, global_id 4156 2024-03-25T19:42:04.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f166f6cf640 10 monclient: _renew_subs 2024-03-25T19:42:04.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f166f6cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:04.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f166f6cf640 1 -- 172.21.15.98:0/3212355169 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166813c010 con 0x7f16680b4700 2024-03-25T19:42:04.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f166f6cf640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:04.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f166f6cf640 10 monclient: _renew_subs 2024-03-25T19:42:04.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f166f6cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:04.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f166f6cf640 1 -- 172.21.15.98:0/3212355169 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1668135bf0 con 0x7f16680b4700 2024-03-25T19:42:04.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.635+0000 7f166f6cf640 1 librados: init done 2024-03-25T19:42:04.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.636+0000 7f166f6cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:04.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.636+0000 7f166f6cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:04.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.636+0000 7f166f6cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:04.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.636+0000 7f166f6cf640 1 -- 172.21.15.98:0/3212355169 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1668138c20 con 0x7f16680b4700 2024-03-25T19:42:04.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.639+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1658003610 con 0x7f16680b4700 2024-03-25T19:42:04.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.639+0000 7f165dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:04.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.639+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1658003800 con 0x7f16680b4700 2024-03-25T19:42:04.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.639+0000 7f165dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.639 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.639+0000 7f165dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:04.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.639+0000 7f165dffb640 10 monclient: dump: 2024-03-25T19:42:04.640 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:04.640 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:04.640 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.640 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.640 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.640 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:04.640 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:04.640 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:04.640 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:04.640 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.639+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1658003db0 con 0x7f16680b4700 2024-03-25T19:42:04.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.639+0000 7f165f7fe640 4 set_mon_vals no callback set 2024-03-25T19:42:04.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.639+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 4228927686 0 0) 0x7f1658006640 con 0x7f16680b4700 2024-03-25T19:42:04.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.639+0000 7f165dffb640 10 client.4156.objecter ms_dispatch 0x7f1668000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:42:04.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.639+0000 7f165dffb640 3 client.4156.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:42:04.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.639+0000 7f165dffb640 3 client.4156.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:42:04.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.639+0000 7f165dffb640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T19:42:04.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.733+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f16580057c0 con 0x7f16680b4700 2024-03-25T19:42:04.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.733+0000 7f165dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:04.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.733+0000 7f165dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:04.766 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:04.766+0000 7fbddaff1040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:42:04.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.805+0000 7f166f6cf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "36f755b4-5184-4bd5-8211-e880c2ae2551", "id": 6}] 2024-03-25T19:42:04.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.805+0000 7f166f6cf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "36f755b4-5184-4bd5-8211-e880c2ae2551", "id": 6}] 2024-03-25T19:42:04.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.805+0000 7f166f6cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:04.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.805+0000 7f166f6cf640 1 -- 172.21.15.98:0/3212355169 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd new", "uuid": "36f755b4-5184-4bd5-8211-e880c2ae2551", "id": 6} v 0) v1 -- 0x7f16680b3800 con 0x7f16680b4700 2024-03-25T19:42:04.813 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:04.814+0000 7f69b922b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:42:04.820 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:04.820+0000 7fbddaff1040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:42:04.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.870+0000 7f165dffb640 1 -- 172.21.15.98:0/3212355169 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "36f755b4-5184-4bd5-8211-e880c2ae2551", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3446873075 0 1113811849) 0x7f1658005960 con 0x7f16680b4700 2024-03-25T19:42:04.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.870+0000 7f165dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "36f755b4-5184-4bd5-8211-e880c2ae2551", "id": 6}] 2024-03-25T19:42:04.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.870+0000 7f165dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:04.870 INFO:teuthology.orchestra.run.smithi098.stdout:6 2024-03-25T19:42:04.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.871+0000 7f166f6cf640 10 librados: watch_flush enter 2024-03-25T19:42:04.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.871+0000 7f166f6cf640 10 librados: watch_flush exit 2024-03-25T19:42:04.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.871+0000 7f166f6cf640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T19:42:04.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.871+0000 7f166f6cf640 10 client.4156.objecter successfully canceled tick 2024-03-25T19:42:04.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.871+0000 7f166f6cf640 10 monclient: shutdown 2024-03-25T19:42:04.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.871+0000 7f166f6cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:04.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.871+0000 7f166f6cf640 1 -- 172.21.15.98:0/3212355169 >> v1:172.21.15.98:6789/0 conn(0x7f16680b4700 legacy=0x7f1668130f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.871+0000 7f166f6cf640 1 -- 172.21.15.98:0/3212355169 shutdown_connections 2024-03-25T19:42:04.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.871+0000 7f166f6cf640 1 -- 172.21.15.98:0/3212355169 >> 172.21.15.98:0/3212355169 conn(0x7f16680668e0 msgr2=0x7f16680b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:04.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.871+0000 7f166f6cf640 1 -- 172.21.15.98:0/3212355169 shutdown_connections 2024-03-25T19:42:04.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.871+0000 7f166f6cf640 1 -- 172.21.15.98:0/3212355169 wait complete. 2024-03-25T19:42:04.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.871+0000 7f166f6cf640 1 librados: shutdown 2024-03-25T19:42:04.881 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --cluster ceph osd new aed342a4-70bb-42a1-a26e-bbc15b6d0234 7 2024-03-25T19:42:04.943 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:04.943+0000 7fbddaff1040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:42:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.980+0000 7fde6647b640 10 monclient: get_monmap_and_config 2024-03-25T19:42:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.980+0000 7fde6647b640 10 monclient: build_initial_monmap 2024-03-25T19:42:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.980+0000 7fde6647b640 10 monclient: monmap: 2024-03-25T19:42:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T19:42:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:42:04.981473+0000 2024-03-25T19:42:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:42:04.981473+0000 2024-03-25T19:42:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:04.980 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:04.981 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:04.981 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:42:04.981 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:42:04.981 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.981+0000 7fde6647b640 1 Processor -- start 2024-03-25T19:42:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.981+0000 7fde6647b640 1 -- start start 2024-03-25T19:42:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.981+0000 7fde6647b640 10 monclient: init 2024-03-25T19:42:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.981+0000 7fde6647b640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:04.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.981+0000 7fde6647b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.981+0000 7fde6647b640 10 monclient(hunting): picked mon.noname-b con 0x7fde6006c2d0 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.981+0000 7fde6647b640 10 monclient(hunting): picked mon.noname-a con 0x7fde6006c6b0 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.981+0000 7fde6647b640 10 monclient(hunting): picked mon.noname-c con 0x7fde60142d90 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.981+0000 7fde6647b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde600b0840 con 0x7fde6006c6b0 2024-03-25T19:42:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.981+0000 7fde6647b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde60143200 con 0x7fde60142d90 2024-03-25T19:42:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.981+0000 7fde6647b640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6006d810 con 0x7fde6006c2d0 2024-03-25T19:42:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.981+0000 7fde6647b640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.981+0000 7fde6647b640 10 monclient(hunting): authenticate will time out at 3960.266567s 2024-03-25T19:42:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.982+0000 7fde64a18640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fde60142d90 0x7fde6014dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51872/0 (socket says 172.21.15.98:51872) 2024-03-25T19:42:04.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.982+0000 7fde64a18640 1 -- 172.21.15.98:0/1591331273 learned_addr learned my addr 172.21.15.98:0/1591331273 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T19:42:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.984+0000 7fde5effd640 1 -- 172.21.15.98:0/1591331273 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313134389 0 0) 0x7fde58001490 con 0x7fde60142d90 2024-03-25T19:42:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.984+0000 7fde5effd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.984+0000 7fde5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.984+0000 7fde5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.984+0000 7fde5effd640 10 monclient(hunting): my global_id is 4159 2024-03-25T19:42:04.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.984+0000 7fde5effd640 1 -- 172.21.15.98:0/1591331273 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde48003520 con 0x7fde60142d90 2024-03-25T19:42:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.985+0000 7fde5effd640 1 -- 172.21.15.98:0/1591331273 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495418245 0 0) 0x7fde6006d810 con 0x7fde6006c2d0 2024-03-25T19:42:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.985+0000 7fde5effd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.985+0000 7fde5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.985+0000 7fde5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.985+0000 7fde5effd640 10 monclient(hunting): my global_id is 4154 2024-03-25T19:42:04.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.985+0000 7fde5effd640 1 -- 172.21.15.98:0/1591331273 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde58001490 con 0x7fde6006c2d0 2024-03-25T19:42:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.987+0000 7fde5effd640 1 -- 172.21.15.98:0/1591331273 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2941665435 0 0) 0x7fde48003520 con 0x7fde60142d90 2024-03-25T19:42:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.987+0000 7fde5effd640 1 -- 172.21.15.98:0/1591331273 >> v1:172.21.15.81:6790/0 conn(0x7fde6006c2d0 legacy=0x7fde600afde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.987+0000 7fde5effd640 1 -- 172.21.15.98:0/1591331273 >> v1:172.21.15.81:6789/0 conn(0x7fde6006c6b0 legacy=0x7fde60142680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:42:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.987+0000 7fde5effd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.987+0000 7fde5effd640 1 monclient: found mon.noname-c 2024-03-25T19:42:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.987+0000 7fde5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:04.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.987+0000 7fde5effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:42:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.987+0000 7fde5effd640 1 -- 172.21.15.98:0/1591331273 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde600ae260 con 0x7fde60142d90 2024-03-25T19:42:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.987+0000 7fde5effd640 10 monclient: _finish_auth 0 2024-03-25T19:42:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.987+0000 7fde5effd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.987+0000 7fde5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.987+0000 7fde6647b640 5 monclient: authenticate success, global_id 4159 2024-03-25T19:42:04.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.987+0000 7fde6647b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:04.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.991+0000 7fde5effd640 1 -- 172.21.15.98:0/1591331273 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fde580019c0 con 0x7fde60142d90 2024-03-25T19:42:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.991+0000 7fde5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.991+0000 7fde5effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.991+0000 7fde5effd640 10 monclient: dump: 2024-03-25T19:42:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:04.991 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.991+0000 7fde5effd640 1 -- 172.21.15.98:0/1591331273 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde58001ba0 con 0x7fde60142d90 2024-03-25T19:42:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.991+0000 7fde5effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.991+0000 7fde5effd640 1 -- 172.21.15.98:0/1591331273 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fde58001f70 con 0x7fde60142d90 2024-03-25T19:42:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.991+0000 7fde5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.991+0000 7fde5effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.991+0000 7fde5effd640 10 monclient: dump: 2024-03-25T19:42:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:04.992 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:04.993 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.991+0000 7fde6647b640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:04.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.991+0000 7fde6647b640 4 set_mon_vals no callback set 2024-03-25T19:42:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.992+0000 7fde6647b640 10 monclient: shutdown 2024-03-25T19:42:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.992+0000 7fde6647b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.992+0000 7fde6647b640 1 -- 172.21.15.98:0/1591331273 >> v1:172.21.15.98:6789/0 conn(0x7fde60142d90 legacy=0x7fde6014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:04.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.992+0000 7fde6647b640 1 -- 172.21.15.98:0/1591331273 shutdown_connections 2024-03-25T19:42:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.992+0000 7fde6647b640 1 -- 172.21.15.98:0/1591331273 >> 172.21.15.98:0/1591331273 conn(0x7fde600b5e40 msgr2=0x7fde600b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.992+0000 7fde6647b640 1 -- 172.21.15.98:0/1591331273 shutdown_connections 2024-03-25T19:42:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.992+0000 7fde6647b640 1 -- 172.21.15.98:0/1591331273 wait complete. 2024-03-25T19:42:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.992+0000 7fde6647b640 10 monclient: build_initial_monmap 2024-03-25T19:42:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.992+0000 7fde6647b640 10 monclient: monmap: 2024-03-25T19:42:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 0 2024-03-25T19:42:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 0.000000 2024-03-25T19:42:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:created 0.000000 2024-03-25T19:42:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 1 2024-03-25T19:42:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:04.996 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:04.997 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:04.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.993+0000 7fde6647b640 1 librados: starting msgr at 2024-03-25T19:42:04.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.993+0000 7fde6647b640 1 librados: starting objecter 2024-03-25T19:42:04.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.993+0000 7fde6647b640 1 Processor -- start 2024-03-25T19:42:04.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.993+0000 7fde6647b640 1 -- start start 2024-03-25T19:42:04.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.993+0000 7fde6647b640 1 librados: setting wanted keys 2024-03-25T19:42:04.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.993+0000 7fde6647b640 1 librados: calling monclient init 2024-03-25T19:42:04.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.993+0000 7fde6647b640 10 monclient: init 2024-03-25T19:42:04.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.994+0000 7fde6647b640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:04.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.994+0000 7fde6647b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:04.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.994+0000 7fde6647b640 10 monclient(hunting): picked mon.noname-a con 0x7fde6006c2d0 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:04.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.994+0000 7fde6647b640 10 monclient(hunting): picked mon.noname-c con 0x7fde6006c6b0 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:04.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.994+0000 7fde6647b640 10 monclient(hunting): picked mon.noname-b con 0x7fde60142d90 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:04.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.994+0000 7fde6647b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde601d7720 con 0x7fde6006c2d0 2024-03-25T19:42:04.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.994+0000 7fde6647b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6013dfc0 con 0x7fde60142d90 2024-03-25T19:42:04.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.994+0000 7fde6647b640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde601d8330 con 0x7fde6006c6b0 2024-03-25T19:42:04.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.994+0000 7fde6647b640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:04.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.994+0000 7fde6647b640 10 monclient(hunting): authenticate will time out at 3960.278839s 2024-03-25T19:42:04.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.994+0000 7fde64a18640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fde60142d90 0x7fde601d4790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.98:51874/0 (socket says 172.21.15.98:51874) 2024-03-25T19:42:04.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.994+0000 7fde64a18640 1 -- 172.21.15.98:0/1451103825 learned_addr learned my addr 172.21.15.98:0/1451103825 (peer_addr_for_me v1:172.21.15.98:0/0) 2024-03-25T19:42:04.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.994+0000 7fde5cff9640 10 client.?.objecter ms_handle_connect 0x7fde60142d90 2024-03-25T19:42:04.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.994+0000 7fde5cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:04.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.995+0000 7fde5cff9640 10 client.?.objecter ms_handle_connect 0x7fde6006c6b0 2024-03-25T19:42:04.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.995+0000 7fde5cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:04.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.996+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498150202 0 0) 0x7fde6013dfc0 con 0x7fde60142d90 2024-03-25T19:42:04.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.996+0000 7fde5cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:04.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.996+0000 7fde5cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:04.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.996+0000 7fde5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:04.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.996+0000 7fde5cff9640 10 monclient(hunting): my global_id is 4162 2024-03-25T19:42:04.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.996+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde4c004eb0 con 0x7fde60142d90 2024-03-25T19:42:04.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.997+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180243224 0 0) 0x7fde601d8330 con 0x7fde6006c6b0 2024-03-25T19:42:04.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.997+0000 7fde5cff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:05.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.997+0000 7fde5cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:05.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.997+0000 7fde5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:05.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.997+0000 7fde5cff9640 10 monclient(hunting): my global_id is 4157 2024-03-25T19:42:05.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:04.997+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde6013dfc0 con 0x7fde6006c6b0 2024-03-25T19:42:05.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.001+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4259742234 0 0) 0x7fde4c004eb0 con 0x7fde60142d90 2024-03-25T19:42:05.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.001+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde601d8330 con 0x7fde60142d90 2024-03-25T19:42:05.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.001+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fde58004b60 con 0x7fde60142d90 2024-03-25T19:42:05.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.001+0000 7fde5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:05.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.001+0000 7fde5cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:05.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.001+0000 7fde5cff9640 10 monclient(hunting): dump: 2024-03-25T19:42:05.001 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:05.001 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:05.001 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:05.001 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:05.001 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:05.001 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:05.001 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:05.001 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:05.001 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:05.001 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:05.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.001+0000 7fde5cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:05.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.002+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 885962998 0 0) 0x7fde6013dfc0 con 0x7fde6006c6b0 2024-03-25T19:42:05.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.002+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde4c004eb0 con 0x7fde6006c6b0 2024-03-25T19:42:05.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.002+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fde54001830 con 0x7fde6006c6b0 2024-03-25T19:42:05.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.002+0000 7fde5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:05.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.002+0000 7fde5cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:05.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.002+0000 7fde5cff9640 10 monclient(hunting): dump: 2024-03-25T19:42:05.003 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:05.003 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:05.003 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:05.003 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:05.003 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:05.003 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:05.003 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:05.003 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:05.003 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:05.003 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:05.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 710931508 0 0) 0x7fde601d8330 con 0x7fde60142d90 2024-03-25T19:42:05.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 >> v1:172.21.15.81:6790/0 conn(0x7fde6006c6b0 legacy=0x7fde6013c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 >> v1:172.21.15.81:6789/0 conn(0x7fde6006c2d0 legacy=0x7fde601d7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde5cff9640 10 monclient: _finish_hunting 0 2024-03-25T19:42:05.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde5cff9640 1 monclient: found mon.b 2024-03-25T19:42:05.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde5cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:05.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde5cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:05.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde601d84c0 con 0x7fde60142d90 2024-03-25T19:42:05.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde5cff9640 10 monclient: _finish_auth 0 2024-03-25T19:42:05.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde5cff9640 10 monclient: _check_auth_tickets 2024-03-25T19:42:05.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde5cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:05.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde6647b640 5 monclient: authenticate success, global_id 4162 2024-03-25T19:42:05.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde6647b640 10 monclient: _renew_subs 2024-03-25T19:42:05.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde6647b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:05.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde6647b640 1 -- 172.21.15.98:0/1451103825 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6013ce10 con 0x7fde60142d90 2024-03-25T19:42:05.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde6647b640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:05.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde6647b640 10 monclient: _renew_subs 2024-03-25T19:42:05.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde6647b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:05.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde6647b640 1 -- 172.21.15.98:0/1451103825 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6013c200 con 0x7fde60142d90 2024-03-25T19:42:05.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.004+0000 7fde6647b640 1 librados: init done 2024-03-25T19:42:05.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.005+0000 7fde6647b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:05.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.005+0000 7fde6647b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:05.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.005+0000 7fde6647b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:05.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.005+0000 7fde6647b640 1 -- 172.21.15.98:0/1451103825 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde60072630 con 0x7fde60142d90 2024-03-25T19:42:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.008+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde58004f70 con 0x7fde60142d90 2024-03-25T19:42:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.008+0000 7fde5cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.008+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fde580050e0 con 0x7fde60142d90 2024-03-25T19:42:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.008+0000 7fde5cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.008+0000 7fde5cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.008+0000 7fde5cff9640 10 monclient: dump: 2024-03-25T19:42:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:epoch 1 2024-03-25T19:42:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:election_strategy: 3 2024-03-25T19:42:05.008 INFO:teuthology.orchestra.run.smithi098.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:05.009 INFO:teuthology.orchestra.run.smithi098.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:05.009 INFO:teuthology.orchestra.run.smithi098.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:05.009 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:42:05.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.008+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fde580055d0 con 0x7fde60142d90 2024-03-25T19:42:05.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.008+0000 7fde5e7fc640 4 set_mon_vals no callback set 2024-03-25T19:42:05.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.008+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 541038512 0 0) 0x7fde58006e00 con 0x7fde60142d90 2024-03-25T19:42:05.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.008+0000 7fde5cff9640 10 client.4162.objecter ms_dispatch 0x7fde60000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:42:05.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.008+0000 7fde5cff9640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:42:05.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.008+0000 7fde5cff9640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:42:05.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.008+0000 7fde5cff9640 20 client.4162.objecter dump_active .. 0 homeless 2024-03-25T19:42:05.101 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:05.101+0000 7f69b922b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:42:05.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.108+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fde580068b0 con 0x7fde60142d90 2024-03-25T19:42:05.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.108+0000 7fde5cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:05.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.108+0000 7fde5cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:05.155 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:05.156+0000 7f69b922b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:42:05.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.177+0000 7fde6647b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "aed342a4-70bb-42a1-a26e-bbc15b6d0234", "id": 7}] 2024-03-25T19:42:05.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.177+0000 7fde6647b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "aed342a4-70bb-42a1-a26e-bbc15b6d0234", "id": 7}] 2024-03-25T19:42:05.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.177+0000 7fde6647b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:05.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.177+0000 7fde6647b640 1 -- 172.21.15.98:0/1451103825 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd new", "uuid": "aed342a4-70bb-42a1-a26e-bbc15b6d0234", "id": 7} v 0) v1 -- 0x7fde6004f730 con 0x7fde60142d90 2024-03-25T19:42:05.282 INFO:tasks.ceph.mgr.y.smithi081.stderr:2024-03-25T19:42:05.283+0000 7f69b922b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:42:05.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.308+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 4034253951 0 0) 0x7fde58006b20 con 0x7fde60142d90 2024-03-25T19:42:05.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.471+0000 7fde5cff9640 1 -- 172.21.15.98:0/1451103825 <== mon.1 v1:172.21.15.98:6789/0 11 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aed342a4-70bb-42a1-a26e-bbc15b6d0234", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3059264671 0 1371665406) 0x7fde58016310 con 0x7fde60142d90 2024-03-25T19:42:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.471+0000 7fde5cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "aed342a4-70bb-42a1-a26e-bbc15b6d0234", "id": 7}] 2024-03-25T19:42:05.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.471+0000 7fde5cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:05.471 INFO:teuthology.orchestra.run.smithi098.stdout:7 2024-03-25T19:42:05.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.472+0000 7fde6647b640 10 librados: watch_flush enter 2024-03-25T19:42:05.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.472+0000 7fde6647b640 10 librados: watch_flush exit 2024-03-25T19:42:05.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.472+0000 7fde6647b640 20 client.4162.objecter shutdown clearing up homeless session... 2024-03-25T19:42:05.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.472+0000 7fde6647b640 10 client.4162.objecter successfully canceled tick 2024-03-25T19:42:05.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.472+0000 7fde6647b640 10 monclient: shutdown 2024-03-25T19:42:05.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.472+0000 7fde6647b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:05.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.472+0000 7fde6647b640 1 -- 172.21.15.98:0/1451103825 >> v1:172.21.15.98:6789/0 conn(0x7fde60142d90 legacy=0x7fde601d4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.472+0000 7fde6647b640 1 -- 172.21.15.98:0/1451103825 shutdown_connections 2024-03-25T19:42:05.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.472+0000 7fde6647b640 1 -- 172.21.15.98:0/1451103825 >> 172.21.15.98:0/1451103825 conn(0x7fde600b5e40 msgr2=0x7fde600b4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:05.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.472+0000 7fde6647b640 1 -- 172.21.15.98:0/1451103825 shutdown_connections 2024-03-25T19:42:05.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.472+0000 7fde6647b640 1 -- 172.21.15.98:0/1451103825 wait complete. 2024-03-25T19:42:05.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-03-25T19:42:05.472+0000 7fde6647b640 1 librados: shutdown 2024-03-25T19:42:05.482 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:42:05.482 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:42:05.482 DEBUG:teuthology.orchestra.run.smithi081:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:42:05.484 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:42:05.484 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:42:05.484 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:42:05.485 DEBUG:teuthology.orchestra.run.smithi081:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:42:05.486 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:42:05.486 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:42:05.486 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:42:05.487 DEBUG:teuthology.orchestra.run.smithi081:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:42:05.488 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:42:05.489 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:42:05.489 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:42:05.489 DEBUG:teuthology.orchestra.run.smithi081:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:42:05.491 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:42:05.491 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:42:05.491 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:42:05.491 DEBUG:teuthology.orchestra.run.smithi098:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:42:05.493 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:42:05.494 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:42:05.494 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:42:05.494 DEBUG:teuthology.orchestra.run.smithi098:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:42:05.496 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:42:05.496 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:42:05.496 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:42:05.496 DEBUG:teuthology.orchestra.run.smithi098:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:42:05.498 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:42:05.498 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:42:05.498 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:42:05.498 DEBUG:teuthology.orchestra.run.smithi098:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:42:05.500 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:42:05.500 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:05.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.653+0000 7f6524179640 10 monclient: get_monmap_and_config 2024-03-25T19:42:05.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.653+0000 7f6524179640 10 monclient: build_initial_monmap 2024-03-25T19:42:05.653 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.653+0000 7f6524179640 10 monclient: monmap: 2024-03-25T19:42:05.653 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T19:42:05.653 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:05.653 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T19:42:05.654557+0000 2024-03-25T19:42:05.653 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T19:42:05.654557+0000 2024-03-25T19:42:05.653 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:05.654 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:05.654 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:05.654 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:42:05.654 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:42:05.654 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T19:42:05.655 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.654+0000 7f6524179640 1 Processor -- start 2024-03-25T19:42:05.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.654+0000 7f6524179640 1 -- start start 2024-03-25T19:42:05.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.654+0000 7f6524179640 10 monclient: init 2024-03-25T19:42:05.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.655+0000 7f6524179640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:05.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.655+0000 7f6524179640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:05.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.655+0000 7f6524179640 10 monclient(hunting): picked mon.noname-b con 0x7f651c0a9920 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:05.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.655+0000 7f6524179640 10 monclient(hunting): picked mon.noname-a con 0x7f651c0b45a0 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:05.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.655+0000 7f6524179640 10 monclient(hunting): picked mon.noname-c con 0x7f651c146450 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:05.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.655+0000 7f6524179640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c06b100 con 0x7f651c0b45a0 2024-03-25T19:42:05.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.655+0000 7f6524179640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c0b2940 con 0x7f651c146450 2024-03-25T19:42:05.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.655+0000 7f6524179640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c13fec0 con 0x7f651c0a9920 2024-03-25T19:42:05.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.655+0000 7f6524179640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:05.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.655+0000 7f6524179640 10 monclient(hunting): authenticate will time out at 3960.737343s 2024-03-25T19:42:05.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.655+0000 7f6521f15640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f651c0a9920 0x7f651c13f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:60266/0 (socket says 172.21.15.81:60266) 2024-03-25T19:42:05.656 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.656+0000 7f6522716640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f651c146450 0x7f651c14dc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:52876/0 (socket says 172.21.15.81:52876) 2024-03-25T19:42:05.657 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.656+0000 7f6522716640 1 -- 172.21.15.81:0/892847084 learned_addr learned my addr 172.21.15.81:0/892847084 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:05.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.780+0000 7f6520f13640 1 -- 172.21.15.81:0/892847084 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241984493 0 0) 0x7f651c0b2940 con 0x7f651c146450 2024-03-25T19:42:05.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.780+0000 7f6520f13640 10 monclient(hunting): _negotiate 2024-03-25T19:42:05.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.780+0000 7f6520f13640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:05.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.780+0000 7f6520f13640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:05.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.780+0000 7f6520f13640 10 monclient(hunting): my global_id is 4171 2024-03-25T19:42:05.780 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.780+0000 7f6520f13640 1 -- 172.21.15.81:0/892847084 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6508003520 con 0x7f651c146450 2024-03-25T19:42:05.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.785+0000 7f6520f13640 1 -- 172.21.15.81:0/892847084 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247430914 0 0) 0x7f651c06b100 con 0x7f651c0b45a0 2024-03-25T19:42:05.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.785+0000 7f6520f13640 10 monclient(hunting): _negotiate 2024-03-25T19:42:05.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.785+0000 7f6520f13640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:05.784 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.785+0000 7f6520f13640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:05.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.785+0000 7f6520f13640 10 monclient(hunting): my global_id is 4173 2024-03-25T19:42:05.785 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.785+0000 7f6520f13640 1 -- 172.21.15.81:0/892847084 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f651c0b2940 con 0x7f651c0b45a0 2024-03-25T19:42:05.790 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.791+0000 7f6520f13640 1 -- 172.21.15.81:0/892847084 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738423236 0 0) 0x7f651c13fec0 con 0x7f651c0a9920 2024-03-25T19:42:05.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.791+0000 7f6520f13640 10 monclient(hunting): _negotiate 2024-03-25T19:42:05.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.791+0000 7f6520f13640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:05.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.791+0000 7f6520f13640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:05.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.791+0000 7f6520f13640 10 monclient(hunting): my global_id is 4166 2024-03-25T19:42:05.791 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.791+0000 7f6520f13640 1 -- 172.21.15.81:0/892847084 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f651c06b100 con 0x7f651c0a9920 2024-03-25T19:42:05.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.799+0000 7f6520f13640 1 -- 172.21.15.81:0/892847084 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2253441631 0 0) 0x7f651c06b100 con 0x7f651c0a9920 2024-03-25T19:42:05.799 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.799+0000 7f6520f13640 1 -- 172.21.15.81:0/892847084 >> v1:172.21.15.98:6789/0 conn(0x7f651c146450 legacy=0x7f651c14dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.799+0000 7f6520f13640 1 -- 172.21.15.81:0/892847084 >> v1:172.21.15.81:6789/0 conn(0x7f651c0b45a0 legacy=0x7f651c145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.800+0000 7f6520f13640 10 monclient: _finish_hunting 0 2024-03-25T19:42:05.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.800+0000 7f6520f13640 1 monclient: found mon.noname-b 2024-03-25T19:42:05.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.800+0000 7f6520f13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:05.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.800+0000 7f6520f13640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:42:05.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.800+0000 7f6520f13640 1 -- 172.21.15.81:0/892847084 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651c140050 con 0x7f651c0a9920 2024-03-25T19:42:05.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.800+0000 7f6520f13640 10 monclient: _finish_auth 0 2024-03-25T19:42:05.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.800+0000 7f6520f13640 10 monclient: _check_auth_tickets 2024-03-25T19:42:05.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.800+0000 7f6520f13640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:05.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.800+0000 7f6520f13640 1 -- 172.21.15.81:0/892847084 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f65180015b0 con 0x7f651c0a9920 2024-03-25T19:42:05.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.800+0000 7f6520f13640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:05.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.800+0000 7f6520f13640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:05.800 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.800+0000 7f6520f13640 10 monclient: dump: 2024-03-25T19:42:05.800 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T19:42:05.801 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:05.801 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:05.801 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:05.801 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:05.801 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:05.801 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:05.801 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:05.801 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:05.801 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T19:42:05.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.800+0000 7f6524179640 5 monclient: authenticate success, global_id 4166 2024-03-25T19:42:05.801 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.800+0000 7f6524179640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:05.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.806+0000 7f6520f13640 1 -- 172.21.15.81:0/892847084 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6518001710 con 0x7f651c0a9920 2024-03-25T19:42:05.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.806+0000 7f6520f13640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:05.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.806+0000 7f6524179640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:05.805 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.806+0000 7f6524179640 4 set_mon_vals no callback set 2024-03-25T19:42:05.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.806+0000 7f6524179640 10 monclient: shutdown 2024-03-25T19:42:05.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.807+0000 7f6520f13640 1 -- 172.21.15.81:0/892847084 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f65180019c0 con 0x7f651c0a9920 2024-03-25T19:42:05.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.807+0000 7f6520f13640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:05.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.807+0000 7f6520f13640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:05.807 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.807+0000 7f6520f13640 10 monclient: dump: 2024-03-25T19:42:05.807 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T19:42:05.807 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:05.807 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:05.807 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:05.807 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:05.808 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:05.808 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:05.808 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:05.808 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:05.808 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T19:42:05.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.807+0000 7f6524179640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:05.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.807+0000 7f6524179640 1 -- 172.21.15.81:0/892847084 >> v1:172.21.15.81:6790/0 conn(0x7f651c0a9920 legacy=0x7f651c13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.807+0000 7f6524179640 1 -- 172.21.15.81:0/892847084 shutdown_connections 2024-03-25T19:42:05.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.807+0000 7f6524179640 1 -- 172.21.15.81:0/892847084 >> 172.21.15.81:0/892847084 conn(0x7f651c0b7f40 msgr2=0x7f651c0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:05.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.807+0000 7f6524179640 1 -- 172.21.15.81:0/892847084 shutdown_connections 2024-03-25T19:42:05.808 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.807+0000 7f6524179640 1 -- 172.21.15.81:0/892847084 wait complete. 2024-03-25T19:42:05.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.808+0000 7f6524179640 10 monclient: build_initial_monmap 2024-03-25T19:42:05.810 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.808+0000 7f6524179640 10 monclient: monmap: 2024-03-25T19:42:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 0 2024-03-25T19:42:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 0.000000 2024-03-25T19:42:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:created 0.000000 2024-03-25T19:42:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 1 2024-03-25T19:42:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:05.811 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T19:42:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.809+0000 7f6524179640 1 librados: starting msgr at 2024-03-25T19:42:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.809+0000 7f6524179640 1 librados: starting objecter 2024-03-25T19:42:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.809+0000 7f6524179640 1 Processor -- start 2024-03-25T19:42:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.809+0000 7f6524179640 1 -- start start 2024-03-25T19:42:05.811 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.809+0000 7f6524179640 1 librados: setting wanted keys 2024-03-25T19:42:05.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.809+0000 7f6524179640 1 librados: calling monclient init 2024-03-25T19:42:05.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.809+0000 7f6524179640 10 monclient: init 2024-03-25T19:42:05.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.810+0000 7f6524179640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:05.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.810+0000 7f6524179640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:05.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.810+0000 7f6524179640 10 monclient(hunting): picked mon.noname-a con 0x7f651c0a9920 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:05.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.810+0000 7f6524179640 10 monclient(hunting): picked mon.noname-c con 0x7f651c0b45a0 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:05.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.811+0000 7f6524179640 10 monclient(hunting): picked mon.noname-b con 0x7f651c146450 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:05.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.811+0000 7f6524179640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c0ac8b0 con 0x7f651c0a9920 2024-03-25T19:42:05.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.811+0000 7f6524179640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c0acb90 con 0x7f651c146450 2024-03-25T19:42:05.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.811+0000 7f6524179640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c1446f0 con 0x7f651c0b45a0 2024-03-25T19:42:05.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.811+0000 7f6524179640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:05.812 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.811+0000 7f6524179640 10 monclient(hunting): authenticate will time out at 3960.893318s 2024-03-25T19:42:05.813 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.814+0000 7f6521714640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f651c0b45a0 0x7f651c1dad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:60276/0 (socket says 172.21.15.81:60276) 2024-03-25T19:42:05.814 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.814+0000 7f6521714640 1 -- 172.21.15.81:0/2143707062 learned_addr learned my addr 172.21.15.81:0/2143707062 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:05.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.819+0000 7f65127fc640 10 client.?.objecter ms_handle_connect 0x7f651c0b45a0 2024-03-25T19:42:05.818 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.819+0000 7f65127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:05.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.821+0000 7f65127fc640 10 client.?.objecter ms_handle_connect 0x7f651c146450 2024-03-25T19:42:05.820 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.821+0000 7f65127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:05.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.825+0000 7f65127fc640 1 -- 172.21.15.81:0/2143707062 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245332334 0 0) 0x7f651c0acb90 con 0x7f651c146450 2024-03-25T19:42:05.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.825+0000 7f65127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:05.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.825+0000 7f65127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:05.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.825+0000 7f65127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:05.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.825+0000 7f65127fc640 10 monclient(hunting): my global_id is 4177 2024-03-25T19:42:05.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.825+0000 7f65127fc640 1 -- 172.21.15.81:0/2143707062 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6500004eb0 con 0x7f651c146450 2024-03-25T19:42:05.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.826+0000 7f65127fc640 1 -- 172.21.15.81:0/2143707062 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965093165 0 0) 0x7f651c1446f0 con 0x7f651c0b45a0 2024-03-25T19:42:05.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.826+0000 7f65127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:05.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.826+0000 7f65127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:05.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.826+0000 7f65127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:05.826 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.826+0000 7f65127fc640 10 monclient(hunting): my global_id is 4172 2024-03-25T19:42:05.827 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.826+0000 7f65127fc640 1 -- 172.21.15.81:0/2143707062 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f651c0acb90 con 0x7f651c0b45a0 2024-03-25T19:42:05.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.833+0000 7f65127fc640 1 -- 172.21.15.81:0/2143707062 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1595769226 0 0) 0x7f6500004eb0 con 0x7f651c146450 2024-03-25T19:42:05.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.833+0000 7f65127fc640 1 -- 172.21.15.81:0/2143707062 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f651c1446f0 con 0x7f651c146450 2024-03-25T19:42:05.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.833+0000 7f65127fc640 1 -- 172.21.15.81:0/2143707062 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f6514001b30 con 0x7f651c146450 2024-03-25T19:42:05.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.833+0000 7f65127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:05.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.833+0000 7f65127fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:05.833 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.833+0000 7f65127fc640 10 monclient(hunting): dump: 2024-03-25T19:42:05.833 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T19:42:05.833 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:05.833 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:05.833 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:05.833 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:05.834 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:05.834 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:05.834 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:05.834 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:05.834 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T19:42:05.834 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.833+0000 7f65127fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:05.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f65127fc640 1 -- 172.21.15.81:0/2143707062 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2572970145 0 0) 0x7f651c1446f0 con 0x7f651c146450 2024-03-25T19:42:05.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f65127fc640 1 -- 172.21.15.81:0/2143707062 >> v1:172.21.15.81:6790/0 conn(0x7f651c0b45a0 legacy=0x7f651c1dad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f65127fc640 1 -- 172.21.15.81:0/2143707062 >> v1:172.21.15.81:6789/0 conn(0x7f651c0a9920 legacy=0x7f651c137170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f65127fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:05.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f65127fc640 1 monclient: found mon.b 2024-03-25T19:42:05.839 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f65127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:05.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f65127fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:05.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f65127fc640 1 -- 172.21.15.81:0/2143707062 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651c1442a0 con 0x7f651c146450 2024-03-25T19:42:05.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f65127fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:05.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f65127fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:05.840 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f65127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:05.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f6524179640 5 monclient: authenticate success, global_id 4177 2024-03-25T19:42:05.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f6524179640 10 monclient: _renew_subs 2024-03-25T19:42:05.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f6524179640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:05.841 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f6524179640 1 -- 172.21.15.81:0/2143707062 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f651c0ab930 con 0x7f651c146450 2024-03-25T19:42:05.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f6524179640 10 client.4177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:05.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f6524179640 10 monclient: _renew_subs 2024-03-25T19:42:05.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f6524179640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:05.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f6524179640 1 -- 172.21.15.81:0/2143707062 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f651c145190 con 0x7f651c146450 2024-03-25T19:42:05.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.839+0000 7f6524179640 1 librados: init done 2024-03-25T19:42:05.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.841+0000 7f6524179640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:05.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.841+0000 7f6524179640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:05.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.841+0000 7f6524179640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:05.842 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.841+0000 7f6524179640 1 -- 172.21.15.81:0/2143707062 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f651c04f730 con 0x7f651c146450 2024-03-25T19:42:05.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.842+0000 7f65127fc640 1 -- 172.21.15.81:0/2143707062 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6514002020 con 0x7f651c146450 2024-03-25T19:42:05.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.842+0000 7f65127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:05.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.842+0000 7f65127fc640 1 -- 172.21.15.81:0/2143707062 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f6514004d00 con 0x7f651c146450 2024-03-25T19:42:05.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.842+0000 7f65127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:05.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.842+0000 7f65127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:05.843 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.842+0000 7f65127fc640 10 monclient: dump: 2024-03-25T19:42:05.844 INFO:teuthology.orchestra.run.smithi081.stderr:epoch 1 2024-03-25T19:42:05.844 INFO:teuthology.orchestra.run.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:05.844 INFO:teuthology.orchestra.run.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:05.844 INFO:teuthology.orchestra.run.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:05.844 INFO:teuthology.orchestra.run.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:05.844 INFO:teuthology.orchestra.run.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:05.844 INFO:teuthology.orchestra.run.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:05.844 INFO:teuthology.orchestra.run.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:05.844 INFO:teuthology.orchestra.run.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:05.844 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T19:42:05.844 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.842+0000 7f6513fff640 4 set_mon_vals no callback set 2024-03-25T19:42:05.856 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:05.857+0000 7f65127fc640 1 -- 172.21.15.81:0/2143707062 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 4034253951 0 0) 0x7f651400fde0 con 0x7f651c146450 2024-03-25T19:42:06.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.029+0000 7f65127fc640 1 -- 172.21.15.81:0/2143707062 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f6514015910 con 0x7f651c146450 2024-03-25T19:42:06.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.029+0000 7f65127fc640 10 client.4177.objecter ms_dispatch 0x7f651c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:06.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.029+0000 7f65127fc640 3 client.4177.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:06.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.029+0000 7f65127fc640 3 client.4177.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:06.029 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.029+0000 7f65127fc640 20 client.4177.objecter dump_active .. 0 homeless 2024-03-25T19:42:06.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.031+0000 7f65127fc640 1 -- 172.21.15.81:0/2143707062 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6514014db0 con 0x7f651c146450 2024-03-25T19:42:06.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.031+0000 7f65127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:06.030 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.031+0000 7f65127fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:06.144 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.145+0000 7f6524179640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:06.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.145+0000 7f6524179640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:06.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.145+0000 7f6524179640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:06.145 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.145+0000 7f6524179640 1 -- 172.21.15.81:0/2143707062 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f651c0b3ba0 con 0x7f651c146450 2024-03-25T19:42:06.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.223+0000 7f65127fc640 1 -- 172.21.15.81:0/2143707062 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f651401d8f0 con 0x7f651c146450 2024-03-25T19:42:06.222 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.223+0000 7f65127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:06.223 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.223+0000 7f65127fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:06.223 INFO:teuthology.orchestra.run.smithi081.stdout: 2024-03-25T19:42:06.223 INFO:teuthology.orchestra.run.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.224+0000 7f6524179640 10 librados: watch_flush enter 2024-03-25T19:42:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.224+0000 7f6524179640 10 librados: watch_flush exit 2024-03-25T19:42:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.224+0000 7f6524179640 20 client.4177.objecter shutdown clearing up homeless session... 2024-03-25T19:42:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.224+0000 7f6524179640 10 client.4177.objecter successfully canceled tick 2024-03-25T19:42:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.224+0000 7f6524179640 10 monclient: shutdown 2024-03-25T19:42:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.224+0000 7f6524179640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:06.224 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.224+0000 7f6524179640 1 -- 172.21.15.81:0/2143707062 >> v1:172.21.15.98:6789/0 conn(0x7f651c146450 legacy=0x7f651c1e30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:06.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.224+0000 7f6524179640 1 -- 172.21.15.81:0/2143707062 shutdown_connections 2024-03-25T19:42:06.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.224+0000 7f6524179640 1 -- 172.21.15.81:0/2143707062 >> 172.21.15.81:0/2143707062 conn(0x7f651c0b7f40 msgr2=0x7f651c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:06.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.224+0000 7f6524179640 1 -- 172.21.15.81:0/2143707062 shutdown_connections 2024-03-25T19:42:06.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.224+0000 7f6524179640 1 -- 172.21.15.81:0/2143707062 wait complete. 2024-03-25T19:42:06.225 INFO:teuthology.orchestra.run.smithi081.stderr:2024-03-25T19:42:06.224+0000 7f6524179640 1 librados: shutdown 2024-03-25T19:42:06.238 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:42:06.238 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:42:07.044 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:07.189 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.189+0000 7fabd2415640 10 monclient: get_monmap_and_config 2024-03-25T19:42:07.189 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.189+0000 7fabd2415640 10 monclient: build_initial_monmap 2024-03-25T19:42:07.189 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.189+0000 7fabd2415640 10 monclient: monmap: 2024-03-25T19:42:07.189 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:42:07.189 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:07.189 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:42:07.190416+0000 2024-03-25T19:42:07.189 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:42:07.190416+0000 2024-03-25T19:42:07.189 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:07.189 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:07.189 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:07.190 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:42:07.190 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:42:07.190 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:07.191 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.190+0000 7fabd2415640 1 Processor -- start 2024-03-25T19:42:07.191 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.190+0000 7fabd2415640 1 -- start start 2024-03-25T19:42:07.191 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.190+0000 7fabd2415640 10 monclient: init 2024-03-25T19:42:07.191 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.191+0000 7fabd2415640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:07.191 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.191+0000 7fabd2415640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:07.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.191+0000 7fabd2415640 10 monclient(hunting): picked mon.noname-c con 0x7fabcc0a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:07.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.191+0000 7fabd2415640 10 monclient(hunting): picked mon.noname-a con 0x7fabcc13f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:07.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.191+0000 7fabd2415640 10 monclient(hunting): picked mon.noname-b con 0x7fabcc1430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:07.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.191+0000 7fabd2415640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc142520 con 0x7fabcc13f710 2024-03-25T19:42:07.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.191+0000 7fabd2415640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc0b28a0 con 0x7fabcc0a98f0 2024-03-25T19:42:07.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.191+0000 7fabd2415640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc0b2a30 con 0x7fabcc1430c0 2024-03-25T19:42:07.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.191+0000 7fabd2415640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:07.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.191+0000 7fabd2415640 10 monclient(hunting): authenticate will time out at 3962.273504s 2024-03-25T19:42:07.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.192+0000 7fabcbfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fabcc0a98f0 0x7fabcc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:52896/0 (socket says 172.21.15.81:52896) 2024-03-25T19:42:07.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.192+0000 7fabcbfff640 1 -- 172.21.15.81:0/789902392 learned_addr learned my addr 172.21.15.81:0/789902392 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:07.196 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.196+0000 7fabcaffd640 1 -- 172.21.15.81:0/789902392 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129617932 0 0) 0x7fabcc0b28a0 con 0x7fabcc0a98f0 2024-03-25T19:42:07.196 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.196+0000 7fabcaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:07.196 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.196+0000 7fabcaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:07.196 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.196+0000 7fabcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:07.197 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.196+0000 7fabcaffd640 10 monclient(hunting): my global_id is 4180 2024-03-25T19:42:07.197 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.196+0000 7fabcaffd640 1 -- 172.21.15.81:0/789902392 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb4003520 con 0x7fabcc0a98f0 2024-03-25T19:42:07.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.200+0000 7fabcaffd640 1 -- 172.21.15.81:0/789902392 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348039338 0 0) 0x7fabcc142520 con 0x7fabcc13f710 2024-03-25T19:42:07.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.200+0000 7fabcaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:07.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.200+0000 7fabcaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:07.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.200+0000 7fabcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:07.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.200+0000 7fabcaffd640 10 monclient(hunting): my global_id is 4176 2024-03-25T19:42:07.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.200+0000 7fabcaffd640 1 -- 172.21.15.81:0/789902392 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabcc0b28a0 con 0x7fabcc13f710 2024-03-25T19:42:07.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.201+0000 7fabcaffd640 1 -- 172.21.15.81:0/789902392 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2760819463 0 0) 0x7fabb4003520 con 0x7fabcc0a98f0 2024-03-25T19:42:07.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.201+0000 7fabcaffd640 1 -- 172.21.15.81:0/789902392 >> v1:172.21.15.81:6790/0 conn(0x7fabcc1430c0 legacy=0x7fabcc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:42:07.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.201+0000 7fabcaffd640 1 -- 172.21.15.81:0/789902392 >> v1:172.21.15.81:6789/0 conn(0x7fabcc13f710 legacy=0x7fabcc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:07.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.201+0000 7fabcaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:07.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.201+0000 7fabcaffd640 1 monclient: found mon.noname-c 2024-03-25T19:42:07.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.201+0000 7fabcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:07.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.201+0000 7fabcaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:42:07.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.201+0000 7fabcaffd640 1 -- 172.21.15.81:0/789902392 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc0b44c0 con 0x7fabcc0a98f0 2024-03-25T19:42:07.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.201+0000 7fabcaffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:07.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.201+0000 7fabcaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:07.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.201+0000 7fabcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:07.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.202+0000 7fabcaffd640 1 -- 172.21.15.81:0/789902392 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fabbc001690 con 0x7fabcc0a98f0 2024-03-25T19:42:07.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.202+0000 7fabcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:07.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.202+0000 7fabcaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:07.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.202+0000 7fabcaffd640 10 monclient: dump: 2024-03-25T19:42:07.203 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:07.203 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:07.203 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:07.203 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:07.203 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:07.203 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:07.204 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:07.204 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:07.204 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:07.204 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:07.206 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.204+0000 7fabcaffd640 1 -- 172.21.15.81:0/789902392 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabbc001870 con 0x7fabcc0a98f0 2024-03-25T19:42:07.206 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.204+0000 7fabcaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:07.206 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.204+0000 7fabcaffd640 1 -- 172.21.15.81:0/789902392 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fabbc001c80 con 0x7fabcc0a98f0 2024-03-25T19:42:07.206 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.204+0000 7fabcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:07.206 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.204+0000 7fabcaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:07.206 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.204+0000 7fabcaffd640 10 monclient: dump: 2024-03-25T19:42:07.206 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:07.206 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:07.206 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:07.206 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:07.206 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:07.206 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:07.207 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:07.207 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:07.207 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:07.207 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:07.207 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.205+0000 7fabd2415640 5 monclient: authenticate success, global_id 4180 2024-03-25T19:42:07.207 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.205+0000 7fabd2415640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:07.207 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.205+0000 7fabd2415640 4 set_mon_vals no callback set 2024-03-25T19:42:07.207 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.206+0000 7fabd2415640 10 monclient: shutdown 2024-03-25T19:42:07.207 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.206+0000 7fabd2415640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:07.207 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.206+0000 7fabd2415640 1 -- 172.21.15.81:0/789902392 >> v1:172.21.15.98:6789/0 conn(0x7fabcc0a98f0 legacy=0x7fabcc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:07.207 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.206+0000 7fabd2415640 1 -- 172.21.15.81:0/789902392 shutdown_connections 2024-03-25T19:42:07.207 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.206+0000 7fabd2415640 1 -- 172.21.15.81:0/789902392 >> 172.21.15.81:0/789902392 conn(0x7fabcc0b7ea0 msgr2=0x7fabcc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:07.207 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.206+0000 7fabd2415640 1 -- 172.21.15.81:0/789902392 shutdown_connections 2024-03-25T19:42:07.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.206+0000 7fabd2415640 1 -- 172.21.15.81:0/789902392 wait complete. 2024-03-25T19:42:07.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.207+0000 7fabd2415640 10 monclient: build_initial_monmap 2024-03-25T19:42:07.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.207+0000 7fabd2415640 10 monclient: monmap: 2024-03-25T19:42:07.210 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:42:07.210 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:07.210 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:42:07.210 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:42:07.210 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:07.211 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:42:07.211 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:07.211 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:07.211 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:07.211 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:07.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.208+0000 7fabd2415640 1 librados: starting msgr at 2024-03-25T19:42:07.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.208+0000 7fabd2415640 1 librados: starting objecter 2024-03-25T19:42:07.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.208+0000 7fabd2415640 1 Processor -- start 2024-03-25T19:42:07.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.208+0000 7fabd2415640 1 -- start start 2024-03-25T19:42:07.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.208+0000 7fabd2415640 1 librados: setting wanted keys 2024-03-25T19:42:07.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.208+0000 7fabd2415640 1 librados: calling monclient init 2024-03-25T19:42:07.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.208+0000 7fabd2415640 10 monclient: init 2024-03-25T19:42:07.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.209+0000 7fabd2415640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:07.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.209+0000 7fabd2415640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:07.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.209+0000 7fabd2415640 10 monclient(hunting): picked mon.noname-c con 0x7fabcc0a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:07.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.209+0000 7fabd2415640 10 monclient(hunting): picked mon.noname-b con 0x7fabcc13f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:07.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.209+0000 7fabd2415640 10 monclient(hunting): picked mon.noname-a con 0x7fabcc1430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:07.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.209+0000 7fabd2415640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc0ac1f0 con 0x7fabcc1430c0 2024-03-25T19:42:07.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.209+0000 7fabd2415640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc0b1570 con 0x7fabcc13f710 2024-03-25T19:42:07.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.209+0000 7fabd2415640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc138ad0 con 0x7fabcc0a98f0 2024-03-25T19:42:07.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.209+0000 7fabd2415640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:07.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.209+0000 7fabd2415640 10 monclient(hunting): authenticate will time out at 3962.291593s 2024-03-25T19:42:07.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.210+0000 7fabd09b2640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fabcc1430c0 0x7fabcc1e6a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:42412/0 (socket says 172.21.15.81:42412) 2024-03-25T19:42:07.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.210+0000 7fabd09b2640 1 -- 172.21.15.81:0/1260397351 learned_addr learned my addr 172.21.15.81:0/1260397351 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:07.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.210+0000 7fabc8ff9640 10 client.?.objecter ms_handle_connect 0x7fabcc1430c0 2024-03-25T19:42:07.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.210+0000 7fabc8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:07.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.210+0000 7fabc8ff9640 10 client.?.objecter ms_handle_connect 0x7fabcc13f710 2024-03-25T19:42:07.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.210+0000 7fabc8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:07.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.213+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847115634 0 0) 0x7fabcc0b1570 con 0x7fabcc13f710 2024-03-25T19:42:07.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.213+0000 7fabc8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:07.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.213+0000 7fabc8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:07.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.213+0000 7fabc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:07.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.213+0000 7fabc8ff9640 10 monclient(hunting): my global_id is 4183 2024-03-25T19:42:07.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.213+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb8004eb0 con 0x7fabcc13f710 2024-03-25T19:42:07.214 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.213+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550287829 0 0) 0x7fabcc0ac1f0 con 0x7fabcc1430c0 2024-03-25T19:42:07.214 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.213+0000 7fabc8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:07.214 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.213+0000 7fabc8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:07.214 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.213+0000 7fabc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:07.214 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.213+0000 7fabc8ff9640 10 monclient(hunting): my global_id is 4179 2024-03-25T19:42:07.214 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.213+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabcc0b1570 con 0x7fabcc1430c0 2024-03-25T19:42:07.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.220+0000 7fabc8ff9640 10 client.?.objecter ms_handle_connect 0x7fabcc0a98f0 2024-03-25T19:42:07.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.220+0000 7fabc8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:07.222 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.222+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2603206568 0 0) 0x7fabb8004eb0 con 0x7fabcc13f710 2024-03-25T19:42:07.222 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.222+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabcc0ac1f0 con 0x7fabcc13f710 2024-03-25T19:42:07.222 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.222+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fabc0001930 con 0x7fabcc13f710 2024-03-25T19:42:07.222 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.222+0000 7fabc8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:07.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.222+0000 7fabc8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:07.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.222+0000 7fabc8ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:07.223 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:07.223 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:07.223 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:07.223 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:07.223 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:07.223 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:07.223 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:07.223 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:07.223 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:07.223 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:07.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.222+0000 7fabc8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:07.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.222+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1588202594 0 0) 0x7fabcc0b1570 con 0x7fabcc1430c0 2024-03-25T19:42:07.224 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.222+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabb8004eb0 con 0x7fabcc1430c0 2024-03-25T19:42:07.224 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.222+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fabc4001a90 con 0x7fabcc1430c0 2024-03-25T19:42:07.224 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.223+0000 7fabc8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:07.224 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.223+0000 7fabc8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:07.224 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.223+0000 7fabc8ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:07.224 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:07.224 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:07.224 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:07.224 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:07.224 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:07.224 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:07.224 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:07.224 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:07.224 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:07.225 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:07.225 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.223+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664011877 0 0) 0x7fabcc138ad0 con 0x7fabcc0a98f0 2024-03-25T19:42:07.225 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.223+0000 7fabc8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:07.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.223+0000 7fabc8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:07.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.223+0000 7fabc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:07.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.223+0000 7fabc8ff9640 10 monclient(hunting): my global_id is 4175 2024-03-25T19:42:07.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.223+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabcc0b1570 con 0x7fabcc0a98f0 2024-03-25T19:42:07.229 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.227+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1622550873 0 0) 0x7fabcc0ac1f0 con 0x7fabcc13f710 2024-03-25T19:42:07.229 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.227+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 >> v1:172.21.15.81:6790/0 conn(0x7fabcc0a98f0 legacy=0x7fabcc0718d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:07.229 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.227+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 >> v1:172.21.15.81:6789/0 conn(0x7fabcc1430c0 legacy=0x7fabcc1e6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:07.229 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.227+0000 7fabc8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:42:07.229 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.227+0000 7fabc8ff9640 1 monclient: found mon.b 2024-03-25T19:42:07.229 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.227+0000 7fabc8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:07.229 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.227+0000 7fabc8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:07.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.227+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc0ac440 con 0x7fabcc13f710 2024-03-25T19:42:07.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.227+0000 7fabc8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:42:07.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.227+0000 7fabc8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:42:07.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.227+0000 7fabc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:07.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.228+0000 7fabd2415640 5 monclient: authenticate success, global_id 4183 2024-03-25T19:42:07.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.228+0000 7fabd2415640 10 monclient: _renew_subs 2024-03-25T19:42:07.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.228+0000 7fabd2415640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:07.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.228+0000 7fabd2415640 1 -- 172.21.15.81:0/1260397351 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabcc0ac720 con 0x7fabcc13f710 2024-03-25T19:42:07.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.228+0000 7fabd2415640 10 client.4183.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:07.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.228+0000 7fabd2415640 10 monclient: _renew_subs 2024-03-25T19:42:07.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.228+0000 7fabd2415640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:07.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.228+0000 7fabd2415640 1 -- 172.21.15.81:0/1260397351 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabcc0ab8b0 con 0x7fabcc13f710 2024-03-25T19:42:07.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.228+0000 7fabd2415640 1 librados: init done 2024-03-25T19:42:07.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.229+0000 7fabd2415640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:07.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.229+0000 7fabd2415640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:07.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.229+0000 7fabd2415640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:07.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.229+0000 7fabd2415640 1 -- 172.21.15.81:0/1260397351 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabcc0b3ef0 con 0x7fabcc13f710 2024-03-25T19:42:07.260 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.260+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabc00035f0 con 0x7fabcc13f710 2024-03-25T19:42:07.260 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.260+0000 7fabc8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:07.260 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.260+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fabc0003900 con 0x7fabcc13f710 2024-03-25T19:42:07.260 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.260+0000 7fabc8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:07.260 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.260+0000 7fabc8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:07.260 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.260+0000 7fabc8ff9640 10 monclient: dump: 2024-03-25T19:42:07.260 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:07.260 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:07.260 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:07.261 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:07.261 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:07.261 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:07.261 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:07.261 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:07.261 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:07.261 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:07.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.260+0000 7fabca7fc640 4 set_mon_vals no callback set 2024-03-25T19:42:07.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.261+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 3) v1 ==== 89499+0+0 (unknown 1652581478 0 0) 0x7fabc0019810 con 0x7fabcc13f710 2024-03-25T19:42:07.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.265+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fabc0053570 con 0x7fabcc13f710 2024-03-25T19:42:07.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.265+0000 7fabc8ff9640 10 client.4183.objecter ms_dispatch 0x7fabcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:07.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.265+0000 7fabc8ff9640 3 client.4183.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:07.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.265+0000 7fabc8ff9640 3 client.4183.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:07.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.265+0000 7fabc8ff9640 20 client.4183.objecter dump_active .. 0 homeless 2024-03-25T19:42:07.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.433+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fabc004eeb0 con 0x7fabcc13f710 2024-03-25T19:42:07.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.433+0000 7fabc8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:07.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.433+0000 7fabc8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:07.556 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.557+0000 7fabd2415640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:07.556 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.557+0000 7fabd2415640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:07.556 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.557+0000 7fabd2415640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:07.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.557+0000 7fabd2415640 1 -- 172.21.15.81:0/1260397351 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabcc0b1720 con 0x7fabcc13f710 2024-03-25T19:42:07.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.583+0000 7fabc8ff9640 1 -- 172.21.15.81:0/1260397351 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fabcc0b1720 con 0x7fabcc13f710 2024-03-25T19:42:07.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.583+0000 7fabc8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:07.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.583+0000 7fabc8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:07.583 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:42:07.583 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:07.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.584+0000 7fabd2415640 10 librados: watch_flush enter 2024-03-25T19:42:07.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.585+0000 7fabd2415640 10 librados: watch_flush exit 2024-03-25T19:42:07.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.585+0000 7fabd2415640 20 client.4183.objecter shutdown clearing up homeless session... 2024-03-25T19:42:07.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.585+0000 7fabd2415640 10 client.4183.objecter successfully canceled tick 2024-03-25T19:42:07.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.585+0000 7fabd2415640 10 monclient: shutdown 2024-03-25T19:42:07.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.585+0000 7fabd2415640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:07.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.585+0000 7fabd2415640 1 -- 172.21.15.81:0/1260397351 >> v1:172.21.15.98:6789/0 conn(0x7fabcc13f710 legacy=0x7fabcc06f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:07.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.585+0000 7fabd2415640 1 -- 172.21.15.81:0/1260397351 shutdown_connections 2024-03-25T19:42:07.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.585+0000 7fabd2415640 1 -- 172.21.15.81:0/1260397351 >> 172.21.15.81:0/1260397351 conn(0x7fabcc0b7ea0 msgr2=0x7fabcc13c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:07.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.585+0000 7fabd2415640 1 -- 172.21.15.81:0/1260397351 shutdown_connections 2024-03-25T19:42:07.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.585+0000 7fabd2415640 1 -- 172.21.15.81:0/1260397351 wait complete. 2024-03-25T19:42:07.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:07.585+0000 7fabd2415640 1 librados: shutdown 2024-03-25T19:42:07.600 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:08.495 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:08.495+0000 7fbd7c44f640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:42:08.541 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:42:08.541+0000 7fbd7c44f640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:42:12.362 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:06.789 35241== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:12.371 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:06.792 35242== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:12.400 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:06.818 35243== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:12.424 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:06.851 35241== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:12.432 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:06.853 35242== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:12.472 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:06.890 35243== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:12.487 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:06.924 35240== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:12.547 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:06.984 35240== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:12.582 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:00:07.003 35595== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:12.665 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:00:07.086 35595== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:12.780 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:00:07.197 35596== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:12.839 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:00:07.256 35596== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:12.881 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:00:07.314 35594== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:12.893 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:00:07.332 35593== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:12.939 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:00:07.373 35594== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:12.951 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:00:07.390 35593== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:12.977 INFO:tasks.ceph.osd.7.smithi098.stderr:2024-03-25T19:42:12.971+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:42:12.981 INFO:tasks.ceph.osd.4.smithi098.stderr:2024-03-25T19:42:12.975+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:42:13.021 INFO:tasks.ceph.osd.6.smithi098.stderr:2024-03-25T19:42:13.015+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:42:13.119 INFO:tasks.ceph.osd.5.smithi098.stderr:2024-03-25T19:42:13.113+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:42:13.209 INFO:tasks.ceph.osd.2.smithi081.stderr:2024-03-25T19:42:13.202+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:42:13.390 INFO:tasks.ceph.osd.3.smithi081.stderr:2024-03-25T19:42:13.384+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:42:13.434 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:07.861 35241== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:13.452 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:07.873 35242== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:13.496 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:07.914 35243== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:13.525 INFO:tasks.ceph.osd.0.smithi081.stderr:2024-03-25T19:42:13.519+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:42:13.529 INFO:tasks.ceph.osd.1.smithi081.stderr:2024-03-25T19:42:13.524+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:42:13.595 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:08.031 35240== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:13.683 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:00:08.104 35595== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:13.801 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:08.228 35241== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:13.803 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:00:08.225 35595== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:13.828 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:08.250 35242== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:13.846 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:00:08.263 35596== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:13.867 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:08.285 35243== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:13.976 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:08.413 35240== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.008 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:00:08.441 35594== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.011 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:00:08.450 35593== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.156 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:08.583 35241== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.158 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:00:08.580 35595== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.190 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:08.611 35242== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.198 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:08.616 35243== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.213 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:00:08.631 35596== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.290 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:08.727 35240== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.381 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:00:08.814 35594== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.384 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:00:08.823 35593== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.406 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:14.442 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:08.870 35241== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.453 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:00:08.875 35595== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.466 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:08.884 35243== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.491 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:08.913 35242== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.495 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.496+0000 7fb5d3fa3640 10 monclient: get_monmap_and_config 2024-03-25T19:42:14.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.496+0000 7fb5d3fa3640 10 monclient: build_initial_monmap 2024-03-25T19:42:14.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.496+0000 7fb5d3fa3640 10 monclient: monmap: 2024-03-25T19:42:14.496 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:42:14.496 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:14.496 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:42:14.497419+0000 2024-03-25T19:42:14.496 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:42:14.497419+0000 2024-03-25T19:42:14.496 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:14.496 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:14.496 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:14.496 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:42:14.496 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:42:14.496 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:14.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.497+0000 7fb5d3fa3640 1 Processor -- start 2024-03-25T19:42:14.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.497+0000 7fb5d3fa3640 1 -- start start 2024-03-25T19:42:14.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.497+0000 7fb5d3fa3640 10 monclient: init 2024-03-25T19:42:14.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.497+0000 7fb5d3fa3640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:14.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.497+0000 7fb5d3fa3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:14.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.497+0000 7fb5d3fa3640 10 monclient(hunting): picked mon.noname-a con 0x7fb5cc06ab80 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:14.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.497+0000 7fb5d3fa3640 10 monclient(hunting): picked mon.noname-c con 0x7fb5cc06af60 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:14.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.497+0000 7fb5d3fa3640 10 monclient(hunting): picked mon.noname-b con 0x7fb5cc0af580 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:14.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.497+0000 7fb5d3fa3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5cc0b04d0 con 0x7fb5cc06ab80 2024-03-25T19:42:14.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.497+0000 7fb5d3fa3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5cc0b1230 con 0x7fb5cc06af60 2024-03-25T19:42:14.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.497+0000 7fb5d3fa3640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5cc0affb0 con 0x7fb5cc0af580 2024-03-25T19:42:14.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.497+0000 7fb5d3fa3640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:14.499 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.497+0000 7fb5d3fa3640 10 monclient(hunting): authenticate will time out at 3969.579726s 2024-03-25T19:42:14.499 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.498+0000 7fb5d1d3f640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb5cc06ab80 0x7fb5cc13f990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:42066/0 (socket says 172.21.15.81:42066) 2024-03-25T19:42:14.499 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.498+0000 7fb5d1d3f640 1 -- 172.21.15.81:0/2847433784 learned_addr learned my addr 172.21.15.81:0/2847433784 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:14.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.500+0000 7fb5d0d3d640 1 -- 172.21.15.81:0/2847433784 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010260393 0 0) 0x7fb5cc0b1230 con 0x7fb5cc06af60 2024-03-25T19:42:14.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.500+0000 7fb5d0d3d640 10 monclient(hunting): _negotiate 2024-03-25T19:42:14.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.500+0000 7fb5d0d3d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:14.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.500+0000 7fb5d0d3d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:14.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.500+0000 7fb5d0d3d640 10 monclient(hunting): my global_id is 4210 2024-03-25T19:42:14.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.500+0000 7fb5d0d3d640 1 -- 172.21.15.81:0/2847433784 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5b8003520 con 0x7fb5cc06af60 2024-03-25T19:42:14.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.501+0000 7fb5d0d3d640 1 -- 172.21.15.81:0/2847433784 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886152834 0 0) 0x7fb5cc0b04d0 con 0x7fb5cc06ab80 2024-03-25T19:42:14.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.501+0000 7fb5d0d3d640 10 monclient(hunting): _negotiate 2024-03-25T19:42:14.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.501+0000 7fb5d0d3d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:14.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.501+0000 7fb5d0d3d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:14.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.501+0000 7fb5d0d3d640 10 monclient(hunting): my global_id is 4206 2024-03-25T19:42:14.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.501+0000 7fb5d0d3d640 1 -- 172.21.15.81:0/2847433784 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5cc0b1230 con 0x7fb5cc06ab80 2024-03-25T19:42:14.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.501+0000 7fb5d0d3d640 1 -- 172.21.15.81:0/2847433784 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130426179 0 0) 0x7fb5cc0affb0 con 0x7fb5cc0af580 2024-03-25T19:42:14.502 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.501+0000 7fb5d0d3d640 10 monclient(hunting): _negotiate 2024-03-25T19:42:14.502 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.501+0000 7fb5d0d3d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:14.502 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.501+0000 7fb5d0d3d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:14.502 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.501+0000 7fb5d0d3d640 10 monclient(hunting): my global_id is 4202 2024-03-25T19:42:14.502 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.501+0000 7fb5d0d3d640 1 -- 172.21.15.81:0/2847433784 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5cc0b04d0 con 0x7fb5cc0af580 2024-03-25T19:42:14.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.503+0000 7fb5d0d3d640 1 -- 172.21.15.81:0/2847433784 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2425259247 0 0) 0x7fb5b8003520 con 0x7fb5cc06af60 2024-03-25T19:42:14.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.504+0000 7fb5d0d3d640 1 -- 172.21.15.81:0/2847433784 >> v1:172.21.15.81:6790/0 conn(0x7fb5cc0af580 legacy=0x7fb5cc14dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.504+0000 7fb5d0d3d640 1 -- 172.21.15.81:0/2847433784 >> v1:172.21.15.81:6789/0 conn(0x7fb5cc06ab80 legacy=0x7fb5cc13f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.504+0000 7fb5d0d3d640 10 monclient: _finish_hunting 0 2024-03-25T19:42:14.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.504+0000 7fb5d0d3d640 1 monclient: found mon.noname-c 2024-03-25T19:42:14.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.504+0000 7fb5d0d3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:14.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.504+0000 7fb5d0d3d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:42:14.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.504+0000 7fb5d0d3d640 1 -- 172.21.15.81:0/2847433784 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5cc0b0140 con 0x7fb5cc06af60 2024-03-25T19:42:14.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.504+0000 7fb5d0d3d640 10 monclient: _finish_auth 0 2024-03-25T19:42:14.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.504+0000 7fb5d0d3d640 10 monclient: _check_auth_tickets 2024-03-25T19:42:14.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.504+0000 7fb5d0d3d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:14.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.504+0000 7fb5d0d3d640 1 -- 172.21.15.81:0/2847433784 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb5bc001690 con 0x7fb5cc06af60 2024-03-25T19:42:14.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.504+0000 7fb5d0d3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:14.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.504+0000 7fb5d0d3d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:14.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.504+0000 7fb5d0d3d640 10 monclient: dump: 2024-03-25T19:42:14.504 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:14.505 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:14.505 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:14.505 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:14.505 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:14.505 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:14.505 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:14.505 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:14.505 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:14.505 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:14.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.504+0000 7fb5d3fa3640 5 monclient: authenticate success, global_id 4210 2024-03-25T19:42:14.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.504+0000 7fb5d3fa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:14.508 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.505+0000 7fb5d0d3d640 1 -- 172.21.15.81:0/2847433784 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5bc0017f0 con 0x7fb5cc06af60 2024-03-25T19:42:14.508 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.505+0000 7fb5d0d3d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:14.508 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.505+0000 7fb5d0d3d640 1 -- 172.21.15.81:0/2847433784 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb5bc001c00 con 0x7fb5cc06af60 2024-03-25T19:42:14.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.505+0000 7fb5d3fa3640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:14.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.505+0000 7fb5d3fa3640 4 set_mon_vals no callback set 2024-03-25T19:42:14.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.505+0000 7fb5d0d3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:14.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.505+0000 7fb5d0d3d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:14.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.505+0000 7fb5d0d3d640 10 monclient: dump: 2024-03-25T19:42:14.509 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:14.509 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:14.509 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:14.509 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:14.509 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:14.509 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:14.509 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:14.509 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:14.509 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:14.510 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:14.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.506+0000 7fb5d3fa3640 10 monclient: shutdown 2024-03-25T19:42:14.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.506+0000 7fb5d3fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:14.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.506+0000 7fb5d3fa3640 1 -- 172.21.15.81:0/2847433784 >> v1:172.21.15.98:6789/0 conn(0x7fb5cc06af60 legacy=0x7fb5cc142370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.506+0000 7fb5d3fa3640 1 -- 172.21.15.81:0/2847433784 shutdown_connections 2024-03-25T19:42:14.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.506+0000 7fb5d3fa3640 1 -- 172.21.15.81:0/2847433784 >> 172.21.15.81:0/2847433784 conn(0x7fb5cc0b4b80 msgr2=0x7fb5cc0b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:14.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.506+0000 7fb5d3fa3640 1 -- 172.21.15.81:0/2847433784 shutdown_connections 2024-03-25T19:42:14.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.506+0000 7fb5d3fa3640 1 -- 172.21.15.81:0/2847433784 wait complete. 2024-03-25T19:42:14.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.507+0000 7fb5d3fa3640 10 monclient: build_initial_monmap 2024-03-25T19:42:14.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.507+0000 7fb5d3fa3640 10 monclient: monmap: 2024-03-25T19:42:14.510 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:42:14.510 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:14.510 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:42:14.510 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:42:14.511 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:14.511 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:42:14.511 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:14.511 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:14.511 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:14.511 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:14.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.507+0000 7fb5d3fa3640 1 librados: starting msgr at 2024-03-25T19:42:14.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.507+0000 7fb5d3fa3640 1 librados: starting objecter 2024-03-25T19:42:14.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.507+0000 7fb5d3fa3640 1 Processor -- start 2024-03-25T19:42:14.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.507+0000 7fb5d3fa3640 1 -- start start 2024-03-25T19:42:14.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.507+0000 7fb5d3fa3640 1 librados: setting wanted keys 2024-03-25T19:42:14.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.507+0000 7fb5d3fa3640 1 librados: calling monclient init 2024-03-25T19:42:14.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.507+0000 7fb5d3fa3640 10 monclient: init 2024-03-25T19:42:14.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5d3fa3640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:14.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5d3fa3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:14.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5d3fa3640 10 monclient(hunting): picked mon.noname-c con 0x7fb5cc06ab80 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:14.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5d3fa3640 10 monclient(hunting): picked mon.noname-b con 0x7fb5cc0af580 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:14.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5d3fa3640 10 monclient(hunting): picked mon.noname-a con 0x7fb5cc1db630 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:14.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5d3fa3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5cc11f0a0 con 0x7fb5cc1db630 2024-03-25T19:42:14.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5d3fa3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5cc11f2f0 con 0x7fb5cc0af580 2024-03-25T19:42:14.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5d3fa3640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5cc1daa00 con 0x7fb5cc06ab80 2024-03-25T19:42:14.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5d3fa3640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:14.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5d3fa3640 10 monclient(hunting): authenticate will time out at 3969.589965s 2024-03-25T19:42:14.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5d2540640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb5cc1db630 0x7fb5cc13beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:42080/0 (socket says 172.21.15.81:42080) 2024-03-25T19:42:14.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5d2540640 1 -- 172.21.15.81:0/1500355544 learned_addr learned my addr 172.21.15.81:0/1500355544 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:14.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5c27fc640 10 client.?.objecter ms_handle_connect 0x7fb5cc1db630 2024-03-25T19:42:14.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:14.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5c27fc640 10 client.?.objecter ms_handle_connect 0x7fb5cc0af580 2024-03-25T19:42:14.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:14.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5c27fc640 10 client.?.objecter ms_handle_connect 0x7fb5cc06ab80 2024-03-25T19:42:14.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.508+0000 7fb5c27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:14.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413371588 0 0) 0x7fb5cc11f2f0 con 0x7fb5cc0af580 2024-03-25T19:42:14.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:14.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:14.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:14.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:42:14.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5b0004eb0 con 0x7fb5cc0af580 2024-03-25T19:42:14.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607324663 0 0) 0x7fb5cc11f0a0 con 0x7fb5cc1db630 2024-03-25T19:42:14.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:14.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:14.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:14.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 10 monclient(hunting): my global_id is 4209 2024-03-25T19:42:14.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5cc11f2f0 con 0x7fb5cc1db630 2024-03-25T19:42:14.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258879796 0 0) 0x7fb5cc1daa00 con 0x7fb5cc06ab80 2024-03-25T19:42:14.515 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:14.515 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:14.515 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:14.515 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 10 monclient(hunting): my global_id is 4205 2024-03-25T19:42:14.515 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.510+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5cc11f0a0 con 0x7fb5cc06ab80 2024-03-25T19:42:14.516 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3000818575 0 0) 0x7fb5b0004eb0 con 0x7fb5cc0af580 2024-03-25T19:42:14.516 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5cc1daa00 con 0x7fb5cc0af580 2024-03-25T19:42:14.516 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb5bc001b00 con 0x7fb5cc0af580 2024-03-25T19:42:14.516 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:14.516 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:14.516 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 10 monclient(hunting): dump: 2024-03-25T19:42:14.516 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:14.516 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:14.516 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:14.516 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:14.517 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:14.517 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:14.517 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:14.517 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:14.517 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:14.517 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:14.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:14.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1337619897 0 0) 0x7fb5cc11f2f0 con 0x7fb5cc1db630 2024-03-25T19:42:14.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5b0004eb0 con 0x7fb5cc1db630 2024-03-25T19:42:14.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb5c4001c10 con 0x7fb5cc1db630 2024-03-25T19:42:14.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:14.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:14.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 10 monclient(hunting): dump: 2024-03-25T19:42:14.517 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:14.518 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:14.518 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:14.518 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:14.518 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:14.518 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:14.518 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:14.518 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:14.518 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:14.518 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:14.518 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2427922705 0 0) 0x7fb5cc11f0a0 con 0x7fb5cc06ab80 2024-03-25T19:42:14.518 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5cc11f2f0 con 0x7fb5cc06ab80 2024-03-25T19:42:14.518 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb5c8001930 con 0x7fb5cc06ab80 2024-03-25T19:42:14.518 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:14.518 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:14.518 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.515+0000 7fb5c27fc640 10 monclient(hunting): dump: 2024-03-25T19:42:14.519 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:14.519 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:14.519 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:14.519 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:14.519 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:14.519 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:14.519 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:14.519 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:14.519 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:14.519 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:14.522 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3340344991 0 0) 0x7fb5cc1daa00 con 0x7fb5cc0af580 2024-03-25T19:42:14.522 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 >> v1:172.21.15.81:6790/0 conn(0x7fb5cc06ab80 legacy=0x7fb5cc11dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 >> v1:172.21.15.81:6789/0 conn(0x7fb5cc1db630 legacy=0x7fb5cc13beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5c27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:14.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5c27fc640 1 monclient: found mon.b 2024-03-25T19:42:14.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:14.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5c27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:14.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5cc144440 con 0x7fb5cc0af580 2024-03-25T19:42:14.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5c27fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:14.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5c27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:14.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:14.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5d3fa3640 5 monclient: authenticate success, global_id 4213 2024-03-25T19:42:14.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5d3fa3640 10 monclient: _renew_subs 2024-03-25T19:42:14.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5d3fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:14.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5d3fa3640 1 -- 172.21.15.81:0/1500355544 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5cc11cf00 con 0x7fb5cc0af580 2024-03-25T19:42:14.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5d3fa3640 10 client.4213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:14.524 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5d3fa3640 10 monclient: _renew_subs 2024-03-25T19:42:14.524 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5d3fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:14.524 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5d3fa3640 1 -- 172.21.15.81:0/1500355544 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5cc13c5c0 con 0x7fb5cc0af580 2024-03-25T19:42:14.524 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.518+0000 7fb5d3fa3640 1 librados: init done 2024-03-25T19:42:14.524 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.519+0000 7fb5d3fa3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:14.524 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.519+0000 7fb5d3fa3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:14.524 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.519+0000 7fb5d3fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:14.524 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.519+0000 7fb5d3fa3640 1 -- 172.21.15.81:0/1500355544 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5cc144630 con 0x7fb5cc0af580 2024-03-25T19:42:14.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.536+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5bc0038a0 con 0x7fb5cc0af580 2024-03-25T19:42:14.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.536+0000 7fb5c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:14.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.536+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb5bc003bb0 con 0x7fb5cc0af580 2024-03-25T19:42:14.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.536+0000 7fb5c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:14.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.536+0000 7fb5c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:14.536 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.536+0000 7fb5c27fc640 10 monclient: dump: 2024-03-25T19:42:14.536 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:14.536 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:14.536 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:14.536 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:14.536 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:14.536 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:14.536 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:14.536 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:14.536 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:14.536 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:14.536 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.536+0000 7fb5c3fff640 4 set_mon_vals no callback set 2024-03-25T19:42:14.537 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.537+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fb5bc019b80 con 0x7fb5cc0af580 2024-03-25T19:42:14.537 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.537+0000 7fb5d1d3f640 10 monclient: get_auth_request con 0x7fb5b006e0a0 auth_method 0 2024-03-25T19:42:14.537 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.537+0000 7fb5c27fc640 10 client.4213.objecter ms_handle_connect 0x7fb5b006e0a0 2024-03-25T19:42:14.537 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.538+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fb5bc053600 con 0x7fb5cc0af580 2024-03-25T19:42:14.537 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.538+0000 7fb5c27fc640 10 client.4213.objecter ms_dispatch 0x7fb5cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:14.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.538+0000 7fb5c27fc640 3 client.4213.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:14.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.538+0000 7fb5c27fc640 3 client.4213.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:14.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.538+0000 7fb5c27fc640 20 client.4213.objecter dump_active .. 0 homeless 2024-03-25T19:42:14.563 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:09.000 35240== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.564 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:00:08.982 35596== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.678 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:00:09.117 35593== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.697 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:00:09.131 35594== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.730 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:09.148 35243== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.731 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:09.157 35241== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.742 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:00:09.163 35595== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.764+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb5bc04f110 con 0x7fb5cc0af580 2024-03-25T19:42:14.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.764+0000 7fb5c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:14.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.764+0000 7fb5c27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:14.780 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:09.202 35242== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.827 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:09.264 35240== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.852 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:00:09.270 35596== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.936+0000 7fb5d3fa3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:14.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.936+0000 7fb5d3fa3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:14.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.936+0000 7fb5d3fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:14.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.936+0000 7fb5d3fa3640 1 -- 172.21.15.81:0/1500355544 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5cc0627b0 con 0x7fb5cc0af580 2024-03-25T19:42:14.942 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:00:09.382 35593== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.951+0000 7fb5c27fc640 1 -- 172.21.15.81:0/1500355544 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fb5bc051040 con 0x7fb5cc0af580 2024-03-25T19:42:14.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.951+0000 7fb5c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:14.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.951+0000 7fb5c27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:14.950 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:42:14.951 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:14.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.953+0000 7fb5d3fa3640 10 librados: watch_flush enter 2024-03-25T19:42:14.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.953+0000 7fb5d3fa3640 10 librados: watch_flush exit 2024-03-25T19:42:14.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.953+0000 7fb5d3fa3640 20 client.4213.objecter shutdown clearing up homeless session... 2024-03-25T19:42:14.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.953+0000 7fb5d3fa3640 10 client.4213.objecter successfully canceled tick 2024-03-25T19:42:14.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.953+0000 7fb5d3fa3640 1 -- 172.21.15.81:0/1500355544 >> v1:172.21.15.98:6800/3980829775 conn(0x7fb5b006e0a0 legacy=0x7fb5b0070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.953+0000 7fb5d3fa3640 10 monclient: shutdown 2024-03-25T19:42:14.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.953+0000 7fb5d3fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:14.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.953+0000 7fb5d3fa3640 1 -- 172.21.15.81:0/1500355544 >> v1:172.21.15.98:6789/0 conn(0x7fb5cc0af580 legacy=0x7fb5cc1daf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:14.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.953+0000 7fb5d3fa3640 1 -- 172.21.15.81:0/1500355544 shutdown_connections 2024-03-25T19:42:14.954 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.953+0000 7fb5d3fa3640 1 -- 172.21.15.81:0/1500355544 >> 172.21.15.81:0/1500355544 conn(0x7fb5cc0b4b80 msgr2=0x7fb5cc0b4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:14.954 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.953+0000 7fb5d3fa3640 1 -- 172.21.15.81:0/1500355544 shutdown_connections 2024-03-25T19:42:14.954 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.953+0000 7fb5d3fa3640 1 -- 172.21.15.81:0/1500355544 wait complete. 2024-03-25T19:42:14.954 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:14.953+0000 7fb5d3fa3640 1 librados: shutdown 2024-03-25T19:42:14.962 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:14.979 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:00:09.412 35594== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.999 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:09.417 35243== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.032 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:09.460 35241== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.038 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:00:09.460 35595== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.075 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:09.497 35242== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.105 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:09.542 35240== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.144 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:00:09.562 35596== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.214 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:00:09.653 35593== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.250 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:00:09.683 35594== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.263 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:09.681 35243== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.320 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:09.748 35241== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.326 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:00:09.748 35595== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.365 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:09.787 35242== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.379 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:09.816 35240== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.436 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:00:09.853 35596== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.483 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:00:09.922 35593== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.516 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:00:09.949 35594== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.533 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:09.951 35243== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.612 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:10.039 35241== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.614 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:00:10.036 35595== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.646 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:10.083 35240== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.666 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:10.088 35242== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.724 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:00:10.142 35596== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.749 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:00:10.188 35593== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.784 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:00:10.217 35594== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.797 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:10.215 35243== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.899 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:10.327 35241== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.902 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:00:10.324 35595== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.920 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:10.357 35240== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.955 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:10.376 35242== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.014 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:00:10.432 35596== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.018 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:00:10.457 35593== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.057 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:00:10.490 35594== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.266 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:10.684 35243== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.285 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:00:10.724 35593== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.316 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:00:10.734 35596== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.321 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:10.739 35243== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.327 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:00:10.761 35594== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.412 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:10.849 35240== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.416 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:00:10.838 35595== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.423 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:10.850 35241== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.467 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:10.904 35240== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.468 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:00:10.890 35595== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.478 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:10.905 35241== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.501 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:10.922 35242== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.555 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:10.976 35242== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.765 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:00:11.204 35593== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.820 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:00:11.259 35593== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.823 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:00:11.257 35594== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.855 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:00:11.273 35596== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.877 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:00:11.311 35594== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:16.910 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:00:11.327 35596== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:21.724 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:16.142 35243== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:21.752 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:16.180 35241== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:21.768 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:21.799 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:00:16.221 35595== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:21.809 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:16.231 35242== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:21.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.897+0000 7fe8b26c3640 10 monclient: get_monmap_and_config 2024-03-25T19:42:21.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.897+0000 7fe8b26c3640 10 monclient: build_initial_monmap 2024-03-25T19:42:21.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.897+0000 7fe8b26c3640 10 monclient: monmap: 2024-03-25T19:42:21.897 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:42:21.897 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:21.897 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:42:21.898038+0000 2024-03-25T19:42:21.897 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:42:21.898038+0000 2024-03-25T19:42:21.897 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:21.897 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:21.897 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:21.897 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:42:21.897 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:42:21.897 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:21.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.897+0000 7fe8b26c3640 1 Processor -- start 2024-03-25T19:42:21.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.897+0000 7fe8b26c3640 1 -- start start 2024-03-25T19:42:21.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.898+0000 7fe8b26c3640 10 monclient: init 2024-03-25T19:42:21.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.898+0000 7fe8b26c3640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:21.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.898+0000 7fe8b26c3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:21.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.898+0000 7fe8b26c3640 10 monclient(hunting): picked mon.noname-b con 0x7fe8ac0b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:21.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.898+0000 7fe8b26c3640 10 monclient(hunting): picked mon.noname-a con 0x7fe8ac0b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:21.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.898+0000 7fe8b26c3640 10 monclient(hunting): picked mon.noname-c con 0x7fe8ac14be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:21.900 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.898+0000 7fe8b26c3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8ac0af940 con 0x7fe8ac0b1730 2024-03-25T19:42:21.900 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.898+0000 7fe8b26c3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8ac0b0a70 con 0x7fe8ac14be90 2024-03-25T19:42:21.900 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.898+0000 7fe8b26c3640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8ac0b0d50 con 0x7fe8ac0b4d60 2024-03-25T19:42:21.900 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.898+0000 7fe8b26c3640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:21.900 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.898+0000 7fe8b26c3640 10 monclient(hunting): authenticate will time out at 3976.980455s 2024-03-25T19:42:21.900 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.899+0000 7fe8ab7fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe8ac0b1730 0x7fe8ac0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:46202/0 (socket says 172.21.15.81:46202) 2024-03-25T19:42:21.900 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.899+0000 7fe8abfff640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fe8ac0b4d60 0x7fe8ac13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:35506/0 (socket says 172.21.15.81:35506) 2024-03-25T19:42:21.900 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.899+0000 7fe8ab7fe640 1 -- 172.21.15.81:0/1776096688 learned_addr learned my addr 172.21.15.81:0/1776096688 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:21.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 1 -- 172.21.15.81:0/1776096688 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690631417 0 0) 0x7fe8ac0af940 con 0x7fe8ac0b1730 2024-03-25T19:42:21.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 10 monclient(hunting): my global_id is 4212 2024-03-25T19:42:21.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 1 -- 172.21.15.81:0/1776096688 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe894003520 con 0x7fe8ac0b1730 2024-03-25T19:42:21.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 1 -- 172.21.15.81:0/1776096688 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150086744 0 0) 0x7fe8ac0b0d50 con 0x7fe8ac0b4d60 2024-03-25T19:42:21.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 10 monclient(hunting): my global_id is 4208 2024-03-25T19:42:21.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 1 -- 172.21.15.81:0/1776096688 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8ac0af940 con 0x7fe8ac0b4d60 2024-03-25T19:42:21.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 1 -- 172.21.15.81:0/1776096688 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381691601 0 0) 0x7fe8ac0b0a70 con 0x7fe8ac14be90 2024-03-25T19:42:21.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 10 monclient(hunting): my global_id is 4216 2024-03-25T19:42:21.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.902+0000 7fe8aaffd640 1 -- 172.21.15.81:0/1776096688 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8ac0b0d50 con 0x7fe8ac14be90 2024-03-25T19:42:21.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.905+0000 7fe8aaffd640 1 -- 172.21.15.81:0/1776096688 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 79747638 0 0) 0x7fe894003520 con 0x7fe8ac0b1730 2024-03-25T19:42:21.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.905+0000 7fe8aaffd640 1 -- 172.21.15.81:0/1776096688 >> v1:172.21.15.81:6790/0 conn(0x7fe8ac0b4d60 legacy=0x7fe8ac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.905+0000 7fe8aaffd640 1 -- 172.21.15.81:0/1776096688 >> v1:172.21.15.98:6789/0 conn(0x7fe8ac14be90 legacy=0x7fe8ac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.905+0000 7fe8aaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:21.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.905+0000 7fe8aaffd640 1 monclient: found mon.noname-a 2024-03-25T19:42:21.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.905+0000 7fe8aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:21.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.905+0000 7fe8aaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:42:21.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.905+0000 7fe8aaffd640 1 -- 172.21.15.81:0/1776096688 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ac0b1030 con 0x7fe8ac0b1730 2024-03-25T19:42:21.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.905+0000 7fe8aaffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:21.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.905+0000 7fe8aaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:21.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.905+0000 7fe8aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:21.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.905+0000 7fe8aaffd640 1 -- 172.21.15.81:0/1776096688 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe8a0001690 con 0x7fe8ac0b1730 2024-03-25T19:42:21.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.905+0000 7fe8aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:21.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.905+0000 7fe8aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:21.908 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.905+0000 7fe8aaffd640 10 monclient: dump: 2024-03-25T19:42:21.908 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:21.908 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:21.908 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:21.908 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:21.908 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:21.908 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:21.908 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:21.908 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:21.908 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:21.908 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:21.908 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.905+0000 7fe8b26c3640 5 monclient: authenticate success, global_id 4212 2024-03-25T19:42:21.908 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.905+0000 7fe8b26c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.907+0000 7fe8aaffd640 1 -- 172.21.15.81:0/1776096688 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8a00017f0 con 0x7fe8ac0b1730 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.907+0000 7fe8aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.907+0000 7fe8b26c3640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.907+0000 7fe8b26c3640 4 set_mon_vals no callback set 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.908+0000 7fe8b26c3640 10 monclient: shutdown 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.908+0000 7fe8b26c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.908+0000 7fe8b26c3640 1 -- 172.21.15.81:0/1776096688 >> v1:172.21.15.81:6789/0 conn(0x7fe8ac0b1730 legacy=0x7fe8ac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.909+0000 7fe8b26c3640 1 -- 172.21.15.81:0/1776096688 shutdown_connections 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.909+0000 7fe8b26c3640 1 -- 172.21.15.81:0/1776096688 >> 172.21.15.81:0/1776096688 conn(0x7fe8ac0617a0 msgr2=0x7fe8ac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.909+0000 7fe8b26c3640 1 -- 172.21.15.81:0/1776096688 shutdown_connections 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.909+0000 7fe8b26c3640 1 -- 172.21.15.81:0/1776096688 wait complete. 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.910+0000 7fe8b26c3640 10 monclient: build_initial_monmap 2024-03-25T19:42:21.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.910+0000 7fe8b26c3640 10 monclient: monmap: 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.911+0000 7fe8b26c3640 1 librados: starting msgr at 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.911+0000 7fe8b26c3640 1 librados: starting objecter 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.911+0000 7fe8b26c3640 1 Processor -- start 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.911+0000 7fe8b26c3640 1 -- start start 2024-03-25T19:42:21.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.911+0000 7fe8b26c3640 1 librados: setting wanted keys 2024-03-25T19:42:21.913 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.911+0000 7fe8b26c3640 1 librados: calling monclient init 2024-03-25T19:42:21.913 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.911+0000 7fe8b26c3640 10 monclient: init 2024-03-25T19:42:21.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.912+0000 7fe8b26c3640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:21.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.912+0000 7fe8b26c3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:21.917 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.912+0000 7fe8b26c3640 10 monclient(hunting): picked mon.noname-b con 0x7fe8ac0b1730 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:21.917 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.912+0000 7fe8b26c3640 10 monclient(hunting): picked mon.noname-a con 0x7fe8ac0b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:21.917 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.912+0000 7fe8b26c3640 10 monclient(hunting): picked mon.noname-c con 0x7fe8ac14be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:21.917 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.912+0000 7fe8b26c3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8ac139e50 con 0x7fe8ac0b4d60 2024-03-25T19:42:21.917 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.912+0000 7fe8b26c3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8ac142800 con 0x7fe8ac0b1730 2024-03-25T19:42:21.917 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.912+0000 7fe8b26c3640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8ac140a90 con 0x7fe8ac14be90 2024-03-25T19:42:21.917 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.912+0000 7fe8b26c3640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:21.917 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.912+0000 7fe8b26c3640 10 monclient(hunting): authenticate will time out at 3976.994477s 2024-03-25T19:42:21.917 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.912+0000 7fe8ab7fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe8ac0b4d60 0x7fe8ac134630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:46218/0 (socket says 172.21.15.81:46218) 2024-03-25T19:42:21.917 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.912+0000 7fe8ab7fe640 1 -- 172.21.15.81:0/779326582 learned_addr learned my addr 172.21.15.81:0/779326582 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:21.917 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.913+0000 7fe8a8ff9640 10 client.?.objecter ms_handle_connect 0x7fe8ac0b4d60 2024-03-25T19:42:21.917 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.913+0000 7fe8a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:21.918 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.913+0000 7fe8a8ff9640 10 client.?.objecter ms_handle_connect 0x7fe8ac14be90 2024-03-25T19:42:21.918 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.913+0000 7fe8a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:21.918 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.913+0000 7fe8a8ff9640 10 client.?.objecter ms_handle_connect 0x7fe8ac0b1730 2024-03-25T19:42:21.918 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.913+0000 7fe8a8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:21.918 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.915+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438518294 0 0) 0x7fe8ac139e50 con 0x7fe8ac0b4d60 2024-03-25T19:42:21.918 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.915+0000 7fe8a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.918 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.915+0000 7fe8a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.918 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.915+0000 7fe8a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.918 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.915+0000 7fe8a8ff9640 10 monclient(hunting): my global_id is 4215 2024-03-25T19:42:21.918 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.915+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe898004eb0 con 0x7fe8ac0b4d60 2024-03-25T19:42:21.918 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.916+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372314375 0 0) 0x7fe8ac140a90 con 0x7fe8ac14be90 2024-03-25T19:42:21.918 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.916+0000 7fe8a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.916+0000 7fe8a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.916+0000 7fe8a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.916+0000 7fe8a8ff9640 10 monclient(hunting): my global_id is 4211 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.916+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8ac139e50 con 0x7fe8ac14be90 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.916+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529607120 0 0) 0x7fe8ac142800 con 0x7fe8ac0b1730 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.916+0000 7fe8a8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.916+0000 7fe8a8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.916+0000 7fe8a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.916+0000 7fe8a8ff9640 10 monclient(hunting): my global_id is 4219 2024-03-25T19:42:21.919 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.916+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8ac140a90 con 0x7fe8ac0b1730 2024-03-25T19:42:21.921 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.921+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1649964915 0 0) 0x7fe898004eb0 con 0x7fe8ac0b4d60 2024-03-25T19:42:21.921 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.921+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8ac142800 con 0x7fe8ac0b4d60 2024-03-25T19:42:21.921 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.921+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe8a0001ce0 con 0x7fe8ac0b4d60 2024-03-25T19:42:21.921 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.921+0000 7fe8a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:21.921 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.921+0000 7fe8a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:21.921 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.921+0000 7fe8a8ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:21.921 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:21.921 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:21.921 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:21.921 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:21.922 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:21.922 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:21.922 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:21.922 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:21.922 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:21.922 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:21.922 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.921+0000 7fe8a8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:21.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.922+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3797089485 0 0) 0x7fe8ac139e50 con 0x7fe8ac14be90 2024-03-25T19:42:21.929 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.922+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe898004eb0 con 0x7fe8ac14be90 2024-03-25T19:42:21.929 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.922+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe8a4001b90 con 0x7fe8ac14be90 2024-03-25T19:42:21.929 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.922+0000 7fe8a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:21.929 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.922+0000 7fe8a8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:21.929 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.922+0000 7fe8a8ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:21.929 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:21.929 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:21.929 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:21.929 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:21.929 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:21.929 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:21.929 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:21.930 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:21.930 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:21.930 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:21.930 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.923+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3336594472 0 0) 0x7fe8ac140a90 con 0x7fe8ac0b1730 2024-03-25T19:42:21.930 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.923+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8ac139e50 con 0x7fe8ac0b1730 2024-03-25T19:42:21.930 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.923+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe89c001930 con 0x7fe8ac0b1730 2024-03-25T19:42:21.930 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.923+0000 7fe8a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:21.930 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.923+0000 7fe8a8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:21.930 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.923+0000 7fe8a8ff9640 10 monclient(hunting): dump: 2024-03-25T19:42:21.930 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:21.930 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:21.930 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:21.930 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:21.930 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:21.931 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:21.931 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:21.931 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:21.931 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:21.931 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:21.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.925+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3867902379 0 0) 0x7fe8ac142800 con 0x7fe8ac0b4d60 2024-03-25T19:42:21.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.925+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 >> v1:172.21.15.81:6790/0 conn(0x7fe8ac14be90 legacy=0x7fe8ac1e7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.925+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 >> v1:172.21.15.98:6789/0 conn(0x7fe8ac0b1730 legacy=0x7fe8ac140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.925+0000 7fe8a8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:42:21.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.925+0000 7fe8a8ff9640 1 monclient: found mon.a 2024-03-25T19:42:21.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.925+0000 7fe8a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:21.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.925+0000 7fe8a8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:42:21.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.925+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ac140c20 con 0x7fe8ac0b4d60 2024-03-25T19:42:21.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.925+0000 7fe8a8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:42:21.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.925+0000 7fe8a8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:42:21.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.925+0000 7fe8a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:21.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.926+0000 7fe8b26c3640 5 monclient: authenticate success, global_id 4215 2024-03-25T19:42:21.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.926+0000 7fe8b26c3640 10 monclient: _renew_subs 2024-03-25T19:42:21.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.926+0000 7fe8b26c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:42:21.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.926+0000 7fe8b26c3640 1 -- 172.21.15.81:0/779326582 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8ac138ed0 con 0x7fe8ac0b4d60 2024-03-25T19:42:21.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.926+0000 7fe8b26c3640 10 client.4215.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:21.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.926+0000 7fe8b26c3640 10 monclient: _renew_subs 2024-03-25T19:42:21.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.926+0000 7fe8b26c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:42:21.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.926+0000 7fe8b26c3640 1 -- 172.21.15.81:0/779326582 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8ac142f00 con 0x7fe8ac0b4d60 2024-03-25T19:42:21.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.926+0000 7fe8b26c3640 1 librados: init done 2024-03-25T19:42:21.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.927+0000 7fe8b26c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:21.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.927+0000 7fe8b26c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:21.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.927+0000 7fe8b26c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:42:21.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:21.927+0000 7fe8b26c3640 1 -- 172.21.15.81:0/779326582 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8ac142990 con 0x7fe8ac0b4d60 2024-03-25T19:42:22.124 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:16.561 35240== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:22.188 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.188+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8a0004a90 con 0x7fe8ac0b4d60 2024-03-25T19:42:22.188 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.188+0000 7fe8a8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:22.188 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.188+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe8a0004e60 con 0x7fe8ac0b4d60 2024-03-25T19:42:22.188 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.188+0000 7fe8a8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:22.188 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.188+0000 7fe8a8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:22.188 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.188+0000 7fe8a8ff9640 10 monclient: dump: 2024-03-25T19:42:22.188 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:22.188 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:22.188 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:22.189 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:22.189 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:22.189 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:22.189 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:22.189 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:22.189 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:22.189 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:22.189 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.188+0000 7fe8aa7fc640 4 set_mon_vals no callback set 2024-03-25T19:42:22.190 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.189+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fe8a0005080 con 0x7fe8ac0b4d60 2024-03-25T19:42:22.190 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.190+0000 7fe8abfff640 10 monclient: get_auth_request con 0x7fe89806e0f0 auth_method 0 2024-03-25T19:42:22.190 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.190+0000 7fe8a8ff9640 10 client.4215.objecter ms_handle_connect 0x7fe89806e0f0 2024-03-25T19:42:22.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.192+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fe8a0054950 con 0x7fe8ac0b4d60 2024-03-25T19:42:22.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.192+0000 7fe8a8ff9640 10 client.4215.objecter ms_dispatch 0x7fe8ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:22.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.192+0000 7fe8a8ff9640 3 client.4215.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:22.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.192+0000 7fe8a8ff9640 3 client.4215.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:22.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.192+0000 7fe8a8ff9640 20 client.4215.objecter dump_active .. 0 homeless 2024-03-25T19:42:22.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.192+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8a0050460 con 0x7fe8ac0b4d60 2024-03-25T19:42:22.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.192+0000 7fe8a8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:22.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.192+0000 7fe8a8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:22.390 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.390+0000 7fe8b26c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:22.390 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.390+0000 7fe8b26c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:22.390 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.390+0000 7fe8b26c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:42:22.390 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.390+0000 7fe8b26c3640 1 -- 172.21.15.81:0/779326582 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8ac0627b0 con 0x7fe8ac0b4d60 2024-03-25T19:42:22.446 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.447+0000 7fe8a8ff9640 1 -- 172.21.15.81:0/779326582 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fe8a0052390 con 0x7fe8ac0b4d60 2024-03-25T19:42:22.447 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.447+0000 7fe8a8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:22.447 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.447+0000 7fe8a8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:22.447 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:42:22.447 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:22.449 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.449+0000 7fe8b26c3640 10 librados: watch_flush enter 2024-03-25T19:42:22.449 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.449+0000 7fe8b26c3640 10 librados: watch_flush exit 2024-03-25T19:42:22.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.449+0000 7fe8b26c3640 20 client.4215.objecter shutdown clearing up homeless session... 2024-03-25T19:42:22.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.449+0000 7fe8b26c3640 10 client.4215.objecter successfully canceled tick 2024-03-25T19:42:22.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.450+0000 7fe8b26c3640 1 -- 172.21.15.81:0/779326582 >> v1:172.21.15.98:6800/3980829775 conn(0x7fe89806e0f0 legacy=0x7fe8980705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:22.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.450+0000 7fe8b26c3640 10 monclient: shutdown 2024-03-25T19:42:22.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.450+0000 7fe8b26c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:22.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.450+0000 7fe8b26c3640 1 -- 172.21.15.81:0/779326582 >> v1:172.21.15.81:6789/0 conn(0x7fe8ac0b4d60 legacy=0x7fe8ac134630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:22.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.450+0000 7fe8b26c3640 1 -- 172.21.15.81:0/779326582 shutdown_connections 2024-03-25T19:42:22.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.450+0000 7fe8b26c3640 1 -- 172.21.15.81:0/779326582 >> 172.21.15.81:0/779326582 conn(0x7fe8ac0617a0 msgr2=0x7fe8ac13c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:22.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.450+0000 7fe8b26c3640 1 -- 172.21.15.81:0/779326582 shutdown_connections 2024-03-25T19:42:22.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.450+0000 7fe8b26c3640 1 -- 172.21.15.81:0/779326582 wait complete. 2024-03-25T19:42:22.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:22.450+0000 7fe8b26c3640 1 librados: shutdown 2024-03-25T19:42:22.460 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:22.496 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:00:16.935 35593== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:22.648 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:00:17.065 35596== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:22.778 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:00:17.211 35594== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:27.196 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:21.623 35241== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:27.200 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:21.627 35241== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:27.250 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:00:21.671 35595== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:27.254 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:00:21.675 35595== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:27.427 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:21.849 35242== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:27.431 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:21.853 35242== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:27.530 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:21.967 35240== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:27.533 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:21.970 35240== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:27.556 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:21.974 35243== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:27.560 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:21.978 35243== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:27.756 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:00:22.194 35593== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:27.759 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:00:22.198 35593== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:28.204 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:00:22.621 35596== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:28.207 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:00:22.625 35596== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:28.281 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:00:22.714 35594== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:28.284 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:00:22.718 35594== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:29.267 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:29.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.366+0000 7fa6fd60d640 10 monclient: get_monmap_and_config 2024-03-25T19:42:29.366 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.366+0000 7fa6fd60d640 10 monclient: build_initial_monmap 2024-03-25T19:42:29.366 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.366+0000 7fa6fd60d640 10 monclient: monmap: 2024-03-25T19:42:29.366 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:42:29.366 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:29.366 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:42:29.367038+0000 2024-03-25T19:42:29.366 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:42:29.367038+0000 2024-03-25T19:42:29.366 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:29.366 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:29.366 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:29.366 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:42:29.366 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:42:29.366 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:29.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.366+0000 7fa6fd60d640 1 Processor -- start 2024-03-25T19:42:29.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.367+0000 7fa6fd60d640 1 -- start start 2024-03-25T19:42:29.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.367+0000 7fa6fd60d640 10 monclient: init 2024-03-25T19:42:29.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.367+0000 7fa6fd60d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:29.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.367+0000 7fa6fd60d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:29.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.367+0000 7fa6fd60d640 10 monclient(hunting): picked mon.noname-c con 0x7fa6f80a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:29.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.367+0000 7fa6fd60d640 10 monclient(hunting): picked mon.noname-b con 0x7fa6f813f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:29.369 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.368+0000 7fa6fd60d640 10 monclient(hunting): picked mon.noname-a con 0x7fa6f81430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:29.369 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.368+0000 7fa6fd60d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f8142520 con 0x7fa6f81430c0 2024-03-25T19:42:29.369 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.368+0000 7fa6fd60d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f80b28a0 con 0x7fa6f80a98f0 2024-03-25T19:42:29.369 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.368+0000 7fa6fd60d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f80b2a30 con 0x7fa6f813f710 2024-03-25T19:42:29.369 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.368+0000 7fa6fd60d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:29.369 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.368+0000 7fa6fd60d640 10 monclient(hunting): authenticate will time out at 3984.450169s 2024-03-25T19:42:29.369 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.368+0000 7fa6f67fc640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fa6f813f710 0x7fa6f81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:42560/0 (socket says 172.21.15.81:42560) 2024-03-25T19:42:29.369 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.368+0000 7fa6f67fc640 1 -- 172.21.15.81:0/45628718 learned_addr learned my addr 172.21.15.81:0/45628718 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:29.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.411+0000 7fa6f5ffb640 1 -- 172.21.15.81:0/45628718 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293830952 0 0) 0x7fa6f80b28a0 con 0x7fa6f80a98f0 2024-03-25T19:42:29.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.411+0000 7fa6f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:29.411 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.411+0000 7fa6f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:29.411 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.411+0000 7fa6f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:29.411 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.411+0000 7fa6f5ffb640 10 monclient(hunting): my global_id is 4222 2024-03-25T19:42:29.411 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.411+0000 7fa6f5ffb640 1 -- 172.21.15.81:0/45628718 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6e0003520 con 0x7fa6f80a98f0 2024-03-25T19:42:29.413 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.413+0000 7fa6f5ffb640 1 -- 172.21.15.81:0/45628718 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106997642 0 0) 0x7fa6f8142520 con 0x7fa6f81430c0 2024-03-25T19:42:29.413 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.413+0000 7fa6f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:29.413 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.413+0000 7fa6f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:29.413 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.413+0000 7fa6f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:29.413 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.413+0000 7fa6f5ffb640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:42:29.413 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.413+0000 7fa6f5ffb640 1 -- 172.21.15.81:0/45628718 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6f80b28a0 con 0x7fa6f81430c0 2024-03-25T19:42:29.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.416+0000 7fa6f5ffb640 1 -- 172.21.15.81:0/45628718 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1001733078 0 0) 0x7fa6e0003520 con 0x7fa6f80a98f0 2024-03-25T19:42:29.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.417+0000 7fa6f5ffb640 1 -- 172.21.15.81:0/45628718 >> v1:172.21.15.81:6790/0 conn(0x7fa6f813f710 legacy=0x7fa6f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.417+0000 7fa6f5ffb640 1 -- 172.21.15.81:0/45628718 >> v1:172.21.15.81:6789/0 conn(0x7fa6f81430c0 legacy=0x7fa6f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.417+0000 7fa6f5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:29.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.417+0000 7fa6f5ffb640 1 monclient: found mon.noname-c 2024-03-25T19:42:29.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.417+0000 7fa6f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:29.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.417+0000 7fa6f5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:42:29.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.417+0000 7fa6f5ffb640 1 -- 172.21.15.81:0/45628718 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f80b44c0 con 0x7fa6f80a98f0 2024-03-25T19:42:29.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.417+0000 7fa6f5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:29.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.417+0000 7fa6f5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:29.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.417+0000 7fa6f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:29.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.417+0000 7fa6fd60d640 5 monclient: authenticate success, global_id 4222 2024-03-25T19:42:29.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.417+0000 7fa6fd60d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:29.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.418+0000 7fa6f5ffb640 1 -- 172.21.15.81:0/45628718 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa6e8001650 con 0x7fa6f80a98f0 2024-03-25T19:42:29.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.418+0000 7fa6f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.418+0000 7fa6f5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:29.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.418+0000 7fa6f5ffb640 10 monclient: dump: 2024-03-25T19:42:29.418 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:29.418 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:29.418 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:29.418 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:29.418 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.418 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:29.418 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:29.418 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:29.418 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:29.418 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:29.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.418+0000 7fa6fd60d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:29.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.419+0000 7fa6f5ffb640 1 -- 172.21.15.81:0/45628718 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6e80018a0 con 0x7fa6f80a98f0 2024-03-25T19:42:29.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.419+0000 7fa6f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:29.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.419+0000 7fa6f5ffb640 1 -- 172.21.15.81:0/45628718 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa6e8001c30 con 0x7fa6f80a98f0 2024-03-25T19:42:29.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.419+0000 7fa6f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.419+0000 7fa6f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:29.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.419+0000 7fa6f5ffb640 10 monclient: dump: 2024-03-25T19:42:29.419 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:29.419 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:29.419 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:29.420 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:29.420 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.420 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:29.420 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:29.420 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:29.420 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:29.420 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:29.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.419+0000 7fa6fd60d640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:29.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.419+0000 7fa6fd60d640 4 set_mon_vals no callback set 2024-03-25T19:42:29.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.420+0000 7fa6fd60d640 10 monclient: shutdown 2024-03-25T19:42:29.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.420+0000 7fa6fd60d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:29.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.420+0000 7fa6fd60d640 1 -- 172.21.15.81:0/45628718 >> v1:172.21.15.98:6789/0 conn(0x7fa6f80a98f0 legacy=0x7fa6f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.420+0000 7fa6fd60d640 1 -- 172.21.15.81:0/45628718 shutdown_connections 2024-03-25T19:42:29.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.420+0000 7fa6fd60d640 1 -- 172.21.15.81:0/45628718 >> 172.21.15.81:0/45628718 conn(0x7fa6f80b7ea0 msgr2=0x7fa6f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:29.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.420+0000 7fa6fd60d640 1 -- 172.21.15.81:0/45628718 shutdown_connections 2024-03-25T19:42:29.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.421+0000 7fa6fd60d640 1 -- 172.21.15.81:0/45628718 wait complete. 2024-03-25T19:42:29.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.421+0000 7fa6fd60d640 10 monclient: build_initial_monmap 2024-03-25T19:42:29.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.421+0000 7fa6fd60d640 10 monclient: monmap: 2024-03-25T19:42:29.421 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:42:29.422 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:29.422 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:42:29.422 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:42:29.422 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:29.422 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:42:29.422 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:29.422 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:29.422 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:29.422 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:29.424 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.422+0000 7fa6fd60d640 1 librados: starting msgr at 2024-03-25T19:42:29.424 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.422+0000 7fa6fd60d640 1 librados: starting objecter 2024-03-25T19:42:29.424 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.422+0000 7fa6fd60d640 1 Processor -- start 2024-03-25T19:42:29.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.422+0000 7fa6fd60d640 1 -- start start 2024-03-25T19:42:29.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.422+0000 7fa6fd60d640 1 librados: setting wanted keys 2024-03-25T19:42:29.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.422+0000 7fa6fd60d640 1 librados: calling monclient init 2024-03-25T19:42:29.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.422+0000 7fa6fd60d640 10 monclient: init 2024-03-25T19:42:29.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.423+0000 7fa6fd60d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:29.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.423+0000 7fa6fd60d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:29.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.423+0000 7fa6fd60d640 10 monclient(hunting): picked mon.noname-c con 0x7fa6f80a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:29.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.423+0000 7fa6fd60d640 10 monclient(hunting): picked mon.noname-b con 0x7fa6f813f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:29.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.424+0000 7fa6fd60d640 10 monclient(hunting): picked mon.noname-a con 0x7fa6f81430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:29.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.424+0000 7fa6fd60d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f8136f30 con 0x7fa6f81430c0 2024-03-25T19:42:29.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.424+0000 7fa6fd60d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f81370c0 con 0x7fa6f813f710 2024-03-25T19:42:29.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.424+0000 7fa6fd60d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f8137250 con 0x7fa6f80a98f0 2024-03-25T19:42:29.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.424+0000 7fa6fd60d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:29.426 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.424+0000 7fa6fd60d640 10 monclient(hunting): authenticate will time out at 3984.506261s 2024-03-25T19:42:29.426 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.424+0000 7fa6f77fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fa6f81430c0 0x7fa6f81e2080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:56464/0 (socket says 172.21.15.81:56464) 2024-03-25T19:42:29.426 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.424+0000 7fa6f77fe640 1 -- 172.21.15.81:0/3456768116 learned_addr learned my addr 172.21.15.81:0/3456768116 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:29.426 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.425+0000 7fa6df7fe640 10 client.?.objecter ms_handle_connect 0x7fa6f81430c0 2024-03-25T19:42:29.426 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.425+0000 7fa6df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:29.426 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.425+0000 7fa6df7fe640 10 client.?.objecter ms_handle_connect 0x7fa6f813f710 2024-03-25T19:42:29.426 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.425+0000 7fa6df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:29.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.427+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442193098 0 0) 0x7fa6f8136f30 con 0x7fa6f81430c0 2024-03-25T19:42:29.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.428+0000 7fa6df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:29.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.428+0000 7fa6df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:29.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.428+0000 7fa6df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:29.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.428+0000 7fa6df7fe640 10 monclient(hunting): my global_id is 4221 2024-03-25T19:42:29.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.428+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6e4004eb0 con 0x7fa6f81430c0 2024-03-25T19:42:29.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.428+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418073641 0 0) 0x7fa6f81370c0 con 0x7fa6f813f710 2024-03-25T19:42:29.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.428+0000 7fa6df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:29.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.428+0000 7fa6df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:29.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.428+0000 7fa6df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:29.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.428+0000 7fa6df7fe640 10 monclient(hunting): my global_id is 4225 2024-03-25T19:42:29.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.428+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6f8136f30 con 0x7fa6f813f710 2024-03-25T19:42:29.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.435+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 813149382 0 0) 0x7fa6f8136f30 con 0x7fa6f813f710 2024-03-25T19:42:29.437 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.435+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6f81370c0 con 0x7fa6f813f710 2024-03-25T19:42:29.437 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.435+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa6ec001880 con 0x7fa6f813f710 2024-03-25T19:42:29.437 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.435+0000 7fa6df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.437 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.435+0000 7fa6df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:29.437 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.435+0000 7fa6df7fe640 10 monclient(hunting): dump: 2024-03-25T19:42:29.437 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:29.438 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:29.438 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:29.438 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:29.438 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.438 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:29.438 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:29.438 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:29.438 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:29.438 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:29.438 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.435+0000 7fa6df7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:29.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.435+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 110117386 0 0) 0x7fa6e4004eb0 con 0x7fa6f81430c0 2024-03-25T19:42:29.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.436+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6f8136f30 con 0x7fa6f81430c0 2024-03-25T19:42:29.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.436+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa6f0001c90 con 0x7fa6f81430c0 2024-03-25T19:42:29.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.436+0000 7fa6df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.436+0000 7fa6df7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:29.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.436+0000 7fa6df7fe640 10 monclient(hunting): dump: 2024-03-25T19:42:29.440 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:29.440 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:29.440 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:29.440 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:29.440 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.440 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:29.440 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:29.440 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:29.440 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:29.440 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:29.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.440+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3476559056 0 0) 0x7fa6f81370c0 con 0x7fa6f813f710 2024-03-25T19:42:29.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.440+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 >> v1:172.21.15.81:6790/0 conn(0x7fa6f80a98f0 legacy=0x7fa6f8139440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.440+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 >> v1:172.21.15.81:6789/0 conn(0x7fa6f81430c0 legacy=0x7fa6f81e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.441+0000 7fa6df7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:29.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.441+0000 7fa6df7fe640 1 monclient: found mon.b 2024-03-25T19:42:29.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.441+0000 7fa6df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:29.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.441+0000 7fa6df7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:29.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.441+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f813c310 con 0x7fa6f813f710 2024-03-25T19:42:29.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.441+0000 7fa6df7fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:29.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.441+0000 7fa6df7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:29.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.441+0000 7fa6df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:29.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.441+0000 7fa6fd60d640 5 monclient: authenticate success, global_id 4225 2024-03-25T19:42:29.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.441+0000 7fa6fd60d640 10 monclient: _renew_subs 2024-03-25T19:42:29.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.441+0000 7fa6fd60d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:29.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.441+0000 7fa6fd60d640 1 -- 172.21.15.81:0/3456768116 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f813c560 con 0x7fa6f813f710 2024-03-25T19:42:29.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.441+0000 7fa6fd60d640 10 client.4225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:29.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.441+0000 7fa6fd60d640 10 monclient: _renew_subs 2024-03-25T19:42:29.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.441+0000 7fa6fd60d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:29.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.441+0000 7fa6fd60d640 1 -- 172.21.15.81:0/3456768116 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f8137c50 con 0x7fa6f813f710 2024-03-25T19:42:29.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.441+0000 7fa6fd60d640 1 librados: init done 2024-03-25T19:42:29.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.442+0000 7fa6dd7fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:29.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.442+0000 7fa6dd7fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:29.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.442+0000 7fa6dd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:29.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.442+0000 7fa6dd7fa640 1 -- 172.21.15.81:0/3456768116 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d8005080 con 0x7fa6f813f710 2024-03-25T19:42:29.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.469+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6ec0047e0 con 0x7fa6f813f710 2024-03-25T19:42:29.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.469+0000 7fa6df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:29.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.469+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa6ec004bf0 con 0x7fa6f813f710 2024-03-25T19:42:29.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.469+0000 7fa6df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:29.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.469+0000 7fa6f57fa640 4 set_mon_vals no callback set 2024-03-25T19:42:29.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.469+0000 7fa6df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:29.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.469+0000 7fa6df7fe640 10 monclient: dump: 2024-03-25T19:42:29.470 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:29.470 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:29.470 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:29.470 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:29.470 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:29.470 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:29.470 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:29.470 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:29.470 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:29.470 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:29.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.471+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fa6ec004e10 con 0x7fa6f813f710 2024-03-25T19:42:29.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.472+0000 7fa6f6ffd640 10 monclient: get_auth_request con 0x7fa6e406de40 auth_method 0 2024-03-25T19:42:29.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.472+0000 7fa6df7fe640 10 client.4225.objecter ms_handle_connect 0x7fa6e406de40 2024-03-25T19:42:29.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.585+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fa6ec0547b0 con 0x7fa6f813f710 2024-03-25T19:42:29.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.585+0000 7fa6df7fe640 10 client.4225.objecter ms_dispatch 0x7fa6f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:29.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.585+0000 7fa6df7fe640 3 client.4225.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:29.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.585+0000 7fa6df7fe640 3 client.4225.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:29.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.585+0000 7fa6df7fe640 20 client.4225.objecter dump_active .. 0 homeless 2024-03-25T19:42:29.770 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.770+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6ec0502e0 con 0x7fa6f813f710 2024-03-25T19:42:29.770 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.770+0000 7fa6df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:29.770 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.770+0000 7fa6df7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:29.961 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.961+0000 7fa6dd7fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:29.961 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.961+0000 7fa6dd7fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:29.961 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.961+0000 7fa6dd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:29.961 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.961+0000 7fa6dd7fa640 1 -- 172.21.15.81:0/3456768116 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6d8005600 con 0x7fa6f813f710 2024-03-25T19:42:29.986 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.985+0000 7fa6df7fe640 1 -- 172.21.15.81:0/3456768116 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fa6ec0521f0 con 0x7fa6f813f710 2024-03-25T19:42:29.986 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.985+0000 7fa6df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:29.986 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.985+0000 7fa6df7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:29.986 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:42:29.986 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:29.987 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.987+0000 7fa6dd7fa640 10 librados: watch_flush enter 2024-03-25T19:42:29.987 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.987+0000 7fa6dd7fa640 10 librados: watch_flush exit 2024-03-25T19:42:29.987 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.987+0000 7fa6dd7fa640 20 client.4225.objecter shutdown clearing up homeless session... 2024-03-25T19:42:29.987 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.987+0000 7fa6dd7fa640 10 client.4225.objecter successfully canceled tick 2024-03-25T19:42:29.987 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.987+0000 7fa6dd7fa640 1 -- 172.21.15.81:0/3456768116 >> v1:172.21.15.98:6800/3980829775 conn(0x7fa6e406de40 legacy=0x7fa6e4070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.988 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.987+0000 7fa6dd7fa640 10 monclient: shutdown 2024-03-25T19:42:29.988 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.987+0000 7fa6dd7fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:29.988 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.987+0000 7fa6dd7fa640 1 -- 172.21.15.81:0/3456768116 >> v1:172.21.15.98:6789/0 conn(0x7fa6f813f710 legacy=0x7fa6f81d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:29.988 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.987+0000 7fa6dd7fa640 1 -- 172.21.15.81:0/3456768116 shutdown_connections 2024-03-25T19:42:29.988 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.987+0000 7fa6dd7fa640 1 -- 172.21.15.81:0/3456768116 >> 172.21.15.81:0/3456768116 conn(0x7fa6f80b7ea0 msgr2=0x7fa6f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:29.988 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.987+0000 7fa6dd7fa640 1 -- 172.21.15.81:0/3456768116 shutdown_connections 2024-03-25T19:42:29.988 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.987+0000 7fa6dd7fa640 1 -- 172.21.15.81:0/3456768116 wait complete. 2024-03-25T19:42:29.988 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:29.987+0000 7fa6dd7fa640 1 librados: shutdown 2024-03-25T19:42:29.997 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:31.470 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:00:25.897 35241== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:31.575 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:00:25.996 35242== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:31.622 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:00:26.043 35595== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:31.854 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:00:26.291 35240== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:32.024 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:00:26.442 35243== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:32.157 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:00:26.596 35593== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:32.498 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:00:26.931 35594== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:32.688 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:00:27.105 35596== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:36.802 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:36.947 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.948+0000 7fb174fa3640 10 monclient: get_monmap_and_config 2024-03-25T19:42:36.948 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.948+0000 7fb174fa3640 10 monclient: build_initial_monmap 2024-03-25T19:42:36.948 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.948+0000 7fb174fa3640 10 monclient: monmap: 2024-03-25T19:42:36.948 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:42:36.948 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:36.948 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:42:36.949258+0000 2024-03-25T19:42:36.948 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:42:36.949258+0000 2024-03-25T19:42:36.948 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:36.948 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:36.948 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:36.948 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:42:36.948 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:42:36.948 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:36.949 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.949+0000 7fb174fa3640 1 Processor -- start 2024-03-25T19:42:36.949 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.949+0000 7fb174fa3640 1 -- start start 2024-03-25T19:42:36.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.949+0000 7fb174fa3640 10 monclient: init 2024-03-25T19:42:36.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.950+0000 7fb174fa3640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:36.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.950+0000 7fb174fa3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:36.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.950+0000 7fb174fa3640 10 monclient(hunting): picked mon.noname-b con 0x7fb1700b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:36.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.950+0000 7fb174fa3640 10 monclient(hunting): picked mon.noname-a con 0x7fb1700b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:36.951 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.950+0000 7fb174fa3640 10 monclient(hunting): picked mon.noname-c con 0x7fb17014be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:36.951 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.950+0000 7fb174fa3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1700af940 con 0x7fb1700b1730 2024-03-25T19:42:36.951 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.950+0000 7fb174fa3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1700b0a70 con 0x7fb17014be90 2024-03-25T19:42:36.951 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.950+0000 7fb174fa3640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1700b0d50 con 0x7fb1700b4d60 2024-03-25T19:42:36.951 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.950+0000 7fb174fa3640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:36.951 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.950+0000 7fb174fa3640 10 monclient(hunting): authenticate will time out at 3992.032345s 2024-03-25T19:42:36.951 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.951+0000 7fb167fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb1700b1730 0x7fb1700b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:56478/0 (socket says 172.21.15.81:56478) 2024-03-25T19:42:36.951 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.951+0000 7fb167fff640 1 -- 172.21.15.81:0/3085526329 learned_addr learned my addr 172.21.15.81:0/3085526329 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:36.955 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.955+0000 7fb16dd9b640 1 -- 172.21.15.81:0/3085526329 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353773359 0 0) 0x7fb1700af940 con 0x7fb1700b1730 2024-03-25T19:42:36.955 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.955+0000 7fb16dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:36.955 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.955+0000 7fb16dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:36.955 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.955+0000 7fb16dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:36.955 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.955+0000 7fb16dd9b640 10 monclient(hunting): my global_id is 4224 2024-03-25T19:42:36.955 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.955+0000 7fb16dd9b640 1 -- 172.21.15.81:0/3085526329 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb15c003520 con 0x7fb1700b1730 2024-03-25T19:42:36.955 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.955+0000 7fb16dd9b640 1 -- 172.21.15.81:0/3085526329 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922996770 0 0) 0x7fb1700b0d50 con 0x7fb1700b4d60 2024-03-25T19:42:36.956 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.955+0000 7fb16dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:36.956 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.955+0000 7fb16dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:36.956 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.955+0000 7fb16dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:36.956 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.955+0000 7fb16dd9b640 10 monclient(hunting): my global_id is 4214 2024-03-25T19:42:36.956 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.955+0000 7fb16dd9b640 1 -- 172.21.15.81:0/3085526329 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1700af940 con 0x7fb1700b4d60 2024-03-25T19:42:36.957 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.956+0000 7fb16dd9b640 1 -- 172.21.15.81:0/3085526329 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122808563 0 0) 0x7fb1700b0a70 con 0x7fb17014be90 2024-03-25T19:42:36.957 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.956+0000 7fb16dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:36.958 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.956+0000 7fb16dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:36.958 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.956+0000 7fb16dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:36.958 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.956+0000 7fb16dd9b640 10 monclient(hunting): my global_id is 4228 2024-03-25T19:42:36.958 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.956+0000 7fb16dd9b640 1 -- 172.21.15.81:0/3085526329 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1700b0d50 con 0x7fb17014be90 2024-03-25T19:42:36.960 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.959+0000 7fb16dd9b640 1 -- 172.21.15.81:0/3085526329 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4127178329 0 0) 0x7fb15c003520 con 0x7fb1700b1730 2024-03-25T19:42:36.960 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.959+0000 7fb16dd9b640 1 -- 172.21.15.81:0/3085526329 >> v1:172.21.15.81:6790/0 conn(0x7fb1700b4d60 legacy=0x7fb17013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:36.960 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.959+0000 7fb16dd9b640 1 -- 172.21.15.81:0/3085526329 >> v1:172.21.15.98:6789/0 conn(0x7fb17014be90 legacy=0x7fb17014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:36.960 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.959+0000 7fb16dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:42:36.960 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.959+0000 7fb16dd9b640 1 monclient: found mon.noname-a 2024-03-25T19:42:36.960 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.959+0000 7fb16dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:36.960 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.959+0000 7fb16dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:42:36.960 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.959+0000 7fb16dd9b640 1 -- 172.21.15.81:0/3085526329 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1700b1030 con 0x7fb1700b1730 2024-03-25T19:42:36.960 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.959+0000 7fb16dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:42:36.960 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.959+0000 7fb16dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:42:36.960 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.959+0000 7fb16dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:36.960 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.960+0000 7fb16dd9b640 1 -- 172.21.15.81:0/3085526329 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb158001650 con 0x7fb1700b1730 2024-03-25T19:42:36.960 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.960+0000 7fb16dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:36.961 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.960+0000 7fb16dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:36.961 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.960+0000 7fb16dd9b640 10 monclient: dump: 2024-03-25T19:42:36.961 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:36.961 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:36.961 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:36.961 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:36.961 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:36.961 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:36.961 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:36.961 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:36.961 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:36.961 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:36.962 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.962+0000 7fb174fa3640 5 monclient: authenticate success, global_id 4224 2024-03-25T19:42:36.962 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.962+0000 7fb174fa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:36.962 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.962+0000 7fb16dd9b640 1 -- 172.21.15.81:0/3085526329 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1580018a0 con 0x7fb1700b1730 2024-03-25T19:42:36.962 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.962+0000 7fb16dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:36.962 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.962+0000 7fb16dd9b640 1 -- 172.21.15.81:0/3085526329 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb158001c30 con 0x7fb1700b1730 2024-03-25T19:42:36.962 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.962+0000 7fb16dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:36.962 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.962+0000 7fb16dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:36.962 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.962+0000 7fb16dd9b640 10 monclient: dump: 2024-03-25T19:42:36.962 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:36.963 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:36.963 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:36.963 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:36.963 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:36.963 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:36.963 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:36.963 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:36.963 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:36.963 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:36.963 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.962+0000 7fb174fa3640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:36.963 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.962+0000 7fb174fa3640 4 set_mon_vals no callback set 2024-03-25T19:42:36.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.963+0000 7fb174fa3640 10 monclient: shutdown 2024-03-25T19:42:36.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.963+0000 7fb174fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:36.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.963+0000 7fb174fa3640 1 -- 172.21.15.81:0/3085526329 >> v1:172.21.15.81:6789/0 conn(0x7fb1700b1730 legacy=0x7fb1700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:36.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.963+0000 7fb174fa3640 1 -- 172.21.15.81:0/3085526329 shutdown_connections 2024-03-25T19:42:36.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.963+0000 7fb174fa3640 1 -- 172.21.15.81:0/3085526329 >> 172.21.15.81:0/3085526329 conn(0x7fb1700617a0 msgr2=0x7fb170061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:36.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.963+0000 7fb174fa3640 1 -- 172.21.15.81:0/3085526329 shutdown_connections 2024-03-25T19:42:36.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.963+0000 7fb174fa3640 1 -- 172.21.15.81:0/3085526329 wait complete. 2024-03-25T19:42:36.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.964+0000 7fb174fa3640 10 monclient: build_initial_monmap 2024-03-25T19:42:36.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.964+0000 7fb174fa3640 10 monclient: monmap: 2024-03-25T19:42:36.966 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:42:36.966 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:36.966 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:42:36.966 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:42:36.966 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:36.966 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:42:36.966 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:36.966 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:36.966 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:36.966 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:36.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.965+0000 7fb174fa3640 1 librados: starting msgr at 2024-03-25T19:42:36.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.965+0000 7fb174fa3640 1 librados: starting objecter 2024-03-25T19:42:36.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.965+0000 7fb174fa3640 1 Processor -- start 2024-03-25T19:42:36.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.965+0000 7fb174fa3640 1 -- start start 2024-03-25T19:42:36.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.965+0000 7fb174fa3640 1 librados: setting wanted keys 2024-03-25T19:42:36.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.965+0000 7fb174fa3640 1 librados: calling monclient init 2024-03-25T19:42:36.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.965+0000 7fb174fa3640 10 monclient: init 2024-03-25T19:42:36.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.966+0000 7fb174fa3640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:36.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.966+0000 7fb174fa3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:36.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.966+0000 7fb174fa3640 10 monclient(hunting): picked mon.noname-c con 0x7fb1700b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:36.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.966+0000 7fb174fa3640 10 monclient(hunting): picked mon.noname-b con 0x7fb1700b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:36.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.966+0000 7fb174fa3640 10 monclient(hunting): picked mon.noname-a con 0x7fb17014be90 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:36.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.967+0000 7fb174fa3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1700aa7b0 con 0x7fb17014be90 2024-03-25T19:42:36.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.967+0000 7fb174fa3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1701dae70 con 0x7fb1700b4d60 2024-03-25T19:42:36.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.967+0000 7fb174fa3640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1701daa90 con 0x7fb1700b1730 2024-03-25T19:42:36.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.967+0000 7fb174fa3640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:36.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.967+0000 7fb174fa3640 10 monclient(hunting): authenticate will time out at 3992.048999s 2024-03-25T19:42:36.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.967+0000 7fb16ed9d640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb17014be90 0x7fb1701e2e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:56492/0 (socket says 172.21.15.81:56492) 2024-03-25T19:42:36.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.967+0000 7fb16ed9d640 1 -- 172.21.15.81:0/3551174642 learned_addr learned my addr 172.21.15.81:0/3551174642 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:36.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.967+0000 7fb166ffd640 10 client.?.objecter ms_handle_connect 0x7fb1700b4d60 2024-03-25T19:42:36.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.967+0000 7fb166ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:36.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.967+0000 7fb166ffd640 10 client.?.objecter ms_handle_connect 0x7fb17014be90 2024-03-25T19:42:36.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.967+0000 7fb166ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:36.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.967+0000 7fb166ffd640 10 client.?.objecter ms_handle_connect 0x7fb1700b1730 2024-03-25T19:42:36.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.967+0000 7fb166ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:36.972 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.970+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855219356 0 0) 0x7fb1701dae70 con 0x7fb1700b4d60 2024-03-25T19:42:36.972 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.970+0000 7fb166ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:36.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.970+0000 7fb166ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:36.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.970+0000 7fb166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:36.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.970+0000 7fb166ffd640 10 monclient(hunting): my global_id is 4231 2024-03-25T19:42:36.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.970+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb150004eb0 con 0x7fb1700b4d60 2024-03-25T19:42:36.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.971+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804416811 0 0) 0x7fb1700aa7b0 con 0x7fb17014be90 2024-03-25T19:42:36.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.971+0000 7fb166ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:36.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.971+0000 7fb166ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:36.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.971+0000 7fb166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:36.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.971+0000 7fb166ffd640 10 monclient(hunting): my global_id is 4227 2024-03-25T19:42:36.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.971+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1701dae70 con 0x7fb17014be90 2024-03-25T19:42:36.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.971+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159594178 0 0) 0x7fb1701daa90 con 0x7fb1700b1730 2024-03-25T19:42:36.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.971+0000 7fb166ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:36.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.971+0000 7fb166ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:36.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.971+0000 7fb166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:36.974 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.971+0000 7fb166ffd640 10 monclient(hunting): my global_id is 4217 2024-03-25T19:42:36.974 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.971+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1700aa7b0 con 0x7fb1700b1730 2024-03-25T19:42:36.975 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.975+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3916017780 0 0) 0x7fb150004eb0 con 0x7fb1700b4d60 2024-03-25T19:42:36.975 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.975+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1701daa90 con 0x7fb1700b4d60 2024-03-25T19:42:36.975 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.975+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb158001bb0 con 0x7fb1700b4d60 2024-03-25T19:42:36.975 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.975+0000 7fb166ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:36.975 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.975+0000 7fb166ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:36.975 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.975+0000 7fb166ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:36.975 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:36.975 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:36.976 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:36.976 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:36.976 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:36.976 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:36.976 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:36.976 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:36.976 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:36.976 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:36.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.975+0000 7fb166ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:36.981 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.978+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1476512474 0 0) 0x7fb1701dae70 con 0x7fb17014be90 2024-03-25T19:42:36.981 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb150004eb0 con 0x7fb17014be90 2024-03-25T19:42:36.981 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb168001b90 con 0x7fb17014be90 2024-03-25T19:42:36.982 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:36.982 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:36.982 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:36.982 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:36.982 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:36.982 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:36.982 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:36.982 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:36.982 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:36.982 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:36.982 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:36.982 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:36.982 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:36.982 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 357579365 0 0) 0x7fb1700aa7b0 con 0x7fb1700b1730 2024-03-25T19:42:36.983 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1701dae70 con 0x7fb1700b1730 2024-03-25T19:42:36.983 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb160001250 con 0x7fb1700b1730 2024-03-25T19:42:36.983 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:36.983 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:36.983 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:36.983 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:36.983 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:36.983 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:36.983 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:36.983 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:36.983 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:36.983 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:36.983 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:36.983 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:36.984 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:36.984 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4030101268 0 0) 0x7fb1701daa90 con 0x7fb1700b4d60 2024-03-25T19:42:36.984 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 >> v1:172.21.15.81:6790/0 conn(0x7fb1700b1730 legacy=0x7fb1701db390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:36.984 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 >> v1:172.21.15.81:6789/0 conn(0x7fb17014be90 legacy=0x7fb1701e2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:36.984 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:36.984 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 1 monclient: found mon.b 2024-03-25T19:42:36.984 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:36.984 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:36.984 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1701dac20 con 0x7fb1700b4d60 2024-03-25T19:42:36.984 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:36.984 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:36.984 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb166ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:36.984 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb174fa3640 5 monclient: authenticate success, global_id 4231 2024-03-25T19:42:36.984 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb174fa3640 10 monclient: _renew_subs 2024-03-25T19:42:36.984 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb174fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:36.985 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb174fa3640 1 -- 172.21.15.81:0/3551174642 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb170142780 con 0x7fb1700b4d60 2024-03-25T19:42:36.985 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb174fa3640 10 client.4231.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:36.985 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb174fa3640 10 monclient: _renew_subs 2024-03-25T19:42:36.985 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb174fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:36.985 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb174fa3640 1 -- 172.21.15.81:0/3551174642 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb170142d40 con 0x7fb1700b4d60 2024-03-25T19:42:36.985 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.979+0000 7fb174fa3640 1 librados: init done 2024-03-25T19:42:36.985 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.981+0000 7fb174fa3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:36.985 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.981+0000 7fb174fa3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:36.985 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.981+0000 7fb174fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:36.985 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:36.981+0000 7fb174fa3640 1 -- 172.21.15.81:0/3551174642 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1700627b0 con 0x7fb1700b4d60 2024-03-25T19:42:37.189 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.190+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb158001d50 con 0x7fb1700b4d60 2024-03-25T19:42:37.190 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.190+0000 7fb166ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:37.190 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.190+0000 7fb16d59a640 4 set_mon_vals no callback set 2024-03-25T19:42:37.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.231+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb158004d90 con 0x7fb1700b4d60 2024-03-25T19:42:37.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.231+0000 7fb166ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:37.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.231+0000 7fb166ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:37.230 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.231+0000 7fb166ffd640 10 monclient: dump: 2024-03-25T19:42:37.230 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:37.231 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:37.231 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:37.231 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:37.231 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:37.231 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:37.231 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:37.231 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:37.231 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:37.231 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:37.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.239+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fb15801ad60 con 0x7fb1700b4d60 2024-03-25T19:42:37.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.240+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fb158084ab0 con 0x7fb1700b4d60 2024-03-25T19:42:37.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.240+0000 7fb166ffd640 10 client.4231.objecter ms_dispatch 0x7fb170000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:37.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.240+0000 7fb166ffd640 3 client.4231.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:37.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.240+0000 7fb166ffd640 3 client.4231.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:37.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.240+0000 7fb166ffd640 20 client.4231.objecter dump_active .. 0 homeless 2024-03-25T19:42:37.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.240+0000 7fb16e59c640 10 monclient: get_auth_request con 0x7fb15006e0f0 auth_method 0 2024-03-25T19:42:37.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.240+0000 7fb166ffd640 10 client.4231.objecter ms_handle_connect 0x7fb15006e0f0 2024-03-25T19:42:37.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.241+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb1580503d0 con 0x7fb1700b4d60 2024-03-25T19:42:37.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.241+0000 7fb166ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:37.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.241+0000 7fb166ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:37.564 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.565+0000 7fb174fa3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:37.564 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.565+0000 7fb174fa3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:37.564 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.565+0000 7fb174fa3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:42:37.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.565+0000 7fb174fa3640 1 -- 172.21.15.81:0/3551174642 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb170142910 con 0x7fb1700b4d60 2024-03-25T19:42:37.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.589+0000 7fb166ffd640 1 -- 172.21.15.81:0/3551174642 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fb158052300 con 0x7fb1700b4d60 2024-03-25T19:42:37.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.589+0000 7fb166ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:37.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.589+0000 7fb166ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:37.589 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:42:37.589 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:37.591 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.591+0000 7fb174fa3640 10 librados: watch_flush enter 2024-03-25T19:42:37.591 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.591+0000 7fb174fa3640 10 librados: watch_flush exit 2024-03-25T19:42:37.591 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.591+0000 7fb174fa3640 20 client.4231.objecter shutdown clearing up homeless session... 2024-03-25T19:42:37.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.591+0000 7fb174fa3640 10 client.4231.objecter successfully canceled tick 2024-03-25T19:42:37.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.591+0000 7fb174fa3640 1 -- 172.21.15.81:0/3551174642 >> v1:172.21.15.98:6800/3980829775 conn(0x7fb15006e0f0 legacy=0x7fb1500705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:37.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.591+0000 7fb174fa3640 10 monclient: shutdown 2024-03-25T19:42:37.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.591+0000 7fb174fa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:37.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.591+0000 7fb174fa3640 1 -- 172.21.15.81:0/3551174642 >> v1:172.21.15.98:6789/0 conn(0x7fb1700b4d60 legacy=0x7fb170137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:37.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.591+0000 7fb174fa3640 1 -- 172.21.15.81:0/3551174642 shutdown_connections 2024-03-25T19:42:37.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.591+0000 7fb174fa3640 1 -- 172.21.15.81:0/3551174642 >> 172.21.15.81:0/3551174642 conn(0x7fb1700617a0 msgr2=0x7fb17013c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:37.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.591+0000 7fb174fa3640 1 -- 172.21.15.81:0/3551174642 shutdown_connections 2024-03-25T19:42:37.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.591+0000 7fb174fa3640 1 -- 172.21.15.81:0/3551174642 wait complete. 2024-03-25T19:42:37.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:37.591+0000 7fb174fa3640 1 librados: shutdown 2024-03-25T19:42:37.601 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:39.812 INFO:tasks.ceph.osd.4.smithi098.stderr:2024-03-25T19:42:39.810+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:42:40.092 INFO:tasks.ceph.osd.2.smithi081.stderr:2024-03-25T19:42:40.089+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:42:40.482 INFO:tasks.ceph.osd.7.smithi098.stderr:2024-03-25T19:42:40.477+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:42:40.656 INFO:tasks.ceph.osd.5.smithi098.stderr:2024-03-25T19:42:40.654+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:42:40.751 INFO:tasks.ceph.osd.1.smithi081.stderr:2024-03-25T19:42:40.680+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:42:40.859 INFO:tasks.ceph.osd.6.smithi098.stderr:2024-03-25T19:42:40.857+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:42:41.108 INFO:tasks.ceph.osd.3.smithi081.stderr:2024-03-25T19:42:41.105+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:42:41.270 INFO:tasks.ceph.osd.0.smithi081.stderr:2024-03-25T19:42:41.266+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:42:44.408 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:44.524 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.525+0000 7fefb42a8640 10 monclient: get_monmap_and_config 2024-03-25T19:42:44.525 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.525+0000 7fefb42a8640 10 monclient: build_initial_monmap 2024-03-25T19:42:44.525 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.525+0000 7fefb42a8640 10 monclient: monmap: 2024-03-25T19:42:44.525 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:42:44.525 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:44.525 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:42:44.526181+0000 2024-03-25T19:42:44.525 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:42:44.526181+0000 2024-03-25T19:42:44.525 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:44.525 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:44.525 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:44.525 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:42:44.525 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:42:44.526 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:44.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.525+0000 7fefb42a8640 1 Processor -- start 2024-03-25T19:42:44.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.526+0000 7fefb42a8640 1 -- start start 2024-03-25T19:42:44.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.526+0000 7fefb42a8640 10 monclient: init 2024-03-25T19:42:44.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.526+0000 7fefb42a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:44.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.526+0000 7fefb42a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:44.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.526+0000 7fefb42a8640 10 monclient(hunting): picked mon.noname-b con 0x7fefac0b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:44.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.526+0000 7fefb42a8640 10 monclient(hunting): picked mon.noname-a con 0x7fefac0b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:44.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.526+0000 7fefb42a8640 10 monclient(hunting): picked mon.noname-c con 0x7fefac14be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:44.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.526+0000 7fefb42a8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac0af940 con 0x7fefac0b1730 2024-03-25T19:42:44.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.526+0000 7fefb42a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac0b0a70 con 0x7fefac14be90 2024-03-25T19:42:44.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.526+0000 7fefb42a8640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac0b0d50 con 0x7fefac0b4d60 2024-03-25T19:42:44.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.526+0000 7fefb42a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:44.528 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.526+0000 7fefb42a8640 10 monclient(hunting): authenticate will time out at 3999.608780s 2024-03-25T19:42:44.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.541+0000 7fefb1843640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fefac0b1730 0x7fefac0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:53528/0 (socket says 172.21.15.81:53528) 2024-03-25T19:42:44.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.541+0000 7fefb1843640 1 -- 172.21.15.81:0/225875867 learned_addr learned my addr 172.21.15.81:0/225875867 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:44.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.559+0000 7fefb1042640 1 -- 172.21.15.81:0/225875867 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393453966 0 0) 0x7fefac0af940 con 0x7fefac0b1730 2024-03-25T19:42:44.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.560+0000 7fefb1042640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.560+0000 7fefb1042640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.560+0000 7fefb1042640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.560+0000 7fefb1042640 10 monclient(hunting): my global_id is 4254 2024-03-25T19:42:44.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.560+0000 7fefb1042640 1 -- 172.21.15.81:0/225875867 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef98003520 con 0x7fefac0b1730 2024-03-25T19:42:44.560 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.561+0000 7fefb1042640 1 -- 172.21.15.81:0/225875867 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601616037 0 0) 0x7fefac0b0a70 con 0x7fefac14be90 2024-03-25T19:42:44.561 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.561+0000 7fefb1042640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.561 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.561+0000 7fefb1042640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.561 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.561+0000 7fefb1042640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.561 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.561+0000 7fefb1042640 10 monclient(hunting): my global_id is 4258 2024-03-25T19:42:44.561 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.561+0000 7fefb1042640 1 -- 172.21.15.81:0/225875867 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefac0af940 con 0x7fefac14be90 2024-03-25T19:42:44.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.566+0000 7fefb1042640 1 -- 172.21.15.81:0/225875867 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35100190 0 0) 0x7fefac0b0d50 con 0x7fefac0b4d60 2024-03-25T19:42:44.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.566+0000 7fefb1042640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.566+0000 7fefb1042640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.566+0000 7fefb1042640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.566+0000 7fefb1042640 10 monclient(hunting): my global_id is 4241 2024-03-25T19:42:44.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.566+0000 7fefb1042640 1 -- 172.21.15.81:0/225875867 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefac0b0a70 con 0x7fefac0b4d60 2024-03-25T19:42:44.581 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.580+0000 7fefb1042640 1 -- 172.21.15.81:0/225875867 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2724514021 0 0) 0x7fef98003520 con 0x7fefac0b1730 2024-03-25T19:42:44.581 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.581+0000 7fefb1042640 1 -- 172.21.15.81:0/225875867 >> v1:172.21.15.81:6790/0 conn(0x7fefac0b4d60 legacy=0x7fefac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.581 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.581+0000 7fefb1042640 1 -- 172.21.15.81:0/225875867 >> v1:172.21.15.98:6789/0 conn(0x7fefac14be90 legacy=0x7fefac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.581 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.581+0000 7fefb1042640 10 monclient: _finish_hunting 0 2024-03-25T19:42:44.581 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.581+0000 7fefb1042640 1 monclient: found mon.noname-a 2024-03-25T19:42:44.581 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.581+0000 7fefb1042640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:44.581 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.581+0000 7fefb1042640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:42:44.581 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.581+0000 7fefb1042640 1 -- 172.21.15.81:0/225875867 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac0b1030 con 0x7fefac0b1730 2024-03-25T19:42:44.581 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.581+0000 7fefb1042640 10 monclient: _finish_auth 0 2024-03-25T19:42:44.581 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.581+0000 7fefb1042640 10 monclient: _check_auth_tickets 2024-03-25T19:42:44.581 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.581+0000 7fefb1042640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:44.581 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.581+0000 7fefb1042640 1 -- 172.21.15.81:0/225875867 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fef9c001690 con 0x7fefac0b1730 2024-03-25T19:42:44.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.581+0000 7fefb1042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:44.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.581+0000 7fefb1042640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:44.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.581+0000 7fefb1042640 10 monclient: dump: 2024-03-25T19:42:44.582 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:44.582 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:44.582 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:44.582 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:44.582 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:44.582 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:44.582 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:44.582 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:44.582 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:44.582 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:44.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.581+0000 7fefb42a8640 5 monclient: authenticate success, global_id 4254 2024-03-25T19:42:44.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.581+0000 7fefb42a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:44.598 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.599+0000 7fefb1042640 1 -- 172.21.15.81:0/225875867 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef9c0017f0 con 0x7fefac0b1730 2024-03-25T19:42:44.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.599+0000 7fefb1042640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:44.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.599+0000 7fefb1042640 1 -- 172.21.15.81:0/225875867 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fef9c001c00 con 0x7fefac0b1730 2024-03-25T19:42:44.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.599+0000 7fefb1042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:44.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.599+0000 7fefb1042640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:44.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.599+0000 7fefb1042640 10 monclient: dump: 2024-03-25T19:42:44.599 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:44.599 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:44.599 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:44.599 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:44.599 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:44.599 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:44.599 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:44.600 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:44.600 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:44.600 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:44.600 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.599+0000 7fefb42a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:44.600 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.599+0000 7fefb42a8640 4 set_mon_vals no callback set 2024-03-25T19:42:44.602 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.600+0000 7fefb42a8640 10 monclient: shutdown 2024-03-25T19:42:44.602 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.600+0000 7fefb42a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:44.602 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.600+0000 7fefb42a8640 1 -- 172.21.15.81:0/225875867 >> v1:172.21.15.81:6789/0 conn(0x7fefac0b1730 legacy=0x7fefac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.602 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.600+0000 7fefb42a8640 1 -- 172.21.15.81:0/225875867 shutdown_connections 2024-03-25T19:42:44.602 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.600+0000 7fefb42a8640 1 -- 172.21.15.81:0/225875867 >> 172.21.15.81:0/225875867 conn(0x7fefac0617a0 msgr2=0x7fefac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:44.602 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.600+0000 7fefb42a8640 1 -- 172.21.15.81:0/225875867 shutdown_connections 2024-03-25T19:42:44.602 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.600+0000 7fefb42a8640 1 -- 172.21.15.81:0/225875867 wait complete. 2024-03-25T19:42:44.602 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.601+0000 7fefb42a8640 10 monclient: build_initial_monmap 2024-03-25T19:42:44.602 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.601+0000 7fefb42a8640 10 monclient: monmap: 2024-03-25T19:42:44.603 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:42:44.603 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:44.603 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:42:44.603 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:42:44.603 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:44.603 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:42:44.603 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:44.603 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:44.603 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:44.603 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:44.603 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.601+0000 7fefb42a8640 1 librados: starting msgr at 2024-03-25T19:42:44.603 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.601+0000 7fefb42a8640 1 librados: starting objecter 2024-03-25T19:42:44.603 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.601+0000 7fefb42a8640 1 Processor -- start 2024-03-25T19:42:44.603 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.601+0000 7fefb42a8640 1 -- start start 2024-03-25T19:42:44.603 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.601+0000 7fefb42a8640 1 librados: setting wanted keys 2024-03-25T19:42:44.603 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.601+0000 7fefb42a8640 1 librados: calling monclient init 2024-03-25T19:42:44.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.601+0000 7fefb42a8640 10 monclient: init 2024-03-25T19:42:44.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.602+0000 7fefb42a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:44.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.602+0000 7fefb42a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:44.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.602+0000 7fefb42a8640 10 monclient(hunting): picked mon.noname-c con 0x7fefac0b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:44.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.602+0000 7fefb42a8640 10 monclient(hunting): picked mon.noname-a con 0x7fefac0b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:44.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.602+0000 7fefb42a8640 10 monclient(hunting): picked mon.noname-b con 0x7fefac14be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:44.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.602+0000 7fefb42a8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac141f10 con 0x7fefac0b4d60 2024-03-25T19:42:44.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.602+0000 7fefb42a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac140da0 con 0x7fefac14be90 2024-03-25T19:42:44.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.602+0000 7fefb42a8640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac138860 con 0x7fefac0b1730 2024-03-25T19:42:44.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.602+0000 7fefb42a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:44.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.602+0000 7fefb42a8640 10 monclient(hunting): authenticate will time out at 3999.684594s 2024-03-25T19:42:44.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.612+0000 7fefb1843640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fefac0b4d60 0x7fefac1d7970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:53572/0 (socket says 172.21.15.81:53572) 2024-03-25T19:42:44.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.612+0000 7fefb1843640 1 -- 172.21.15.81:0/3231988985 learned_addr learned my addr 172.21.15.81:0/3231988985 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:44.612 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.613+0000 7fefa2ffd640 10 client.?.objecter ms_handle_connect 0x7fefac0b4d60 2024-03-25T19:42:44.612 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.613+0000 7fefa2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:44.614 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.614+0000 7fefa2ffd640 10 client.?.objecter ms_handle_connect 0x7fefac14be90 2024-03-25T19:42:44.614 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.614+0000 7fefa2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:44.617 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.618+0000 7fefa2ffd640 10 client.?.objecter ms_handle_connect 0x7fefac0b1730 2024-03-25T19:42:44.617 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.618+0000 7fefa2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:44.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.626+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 511089654 0 0) 0x7fefac138860 con 0x7fefac0b1730 2024-03-25T19:42:44.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.626+0000 7fefa2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.626+0000 7fefa2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.626+0000 7fefa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.626+0000 7fefa2ffd640 10 monclient(hunting): my global_id is 4244 2024-03-25T19:42:44.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.626+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef90004eb0 con 0x7fefac0b1730 2024-03-25T19:42:44.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.629+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507318987 0 0) 0x7fefac141f10 con 0x7fefac0b4d60 2024-03-25T19:42:44.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.629+0000 7fefa2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.629+0000 7fefa2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.629+0000 7fefa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.629+0000 7fefa2ffd640 10 monclient(hunting): my global_id is 4257 2024-03-25T19:42:44.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.629+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefac138860 con 0x7fefac0b4d60 2024-03-25T19:42:44.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.631+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147391189 0 0) 0x7fefac140da0 con 0x7fefac14be90 2024-03-25T19:42:44.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.631+0000 7fefa2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:44.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.631+0000 7fefa2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:44.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.631+0000 7fefa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:44.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.631+0000 7fefa2ffd640 10 monclient(hunting): my global_id is 4261 2024-03-25T19:42:44.631 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.631+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefac141f10 con 0x7fefac14be90 2024-03-25T19:42:44.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.648+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 34166697 0 0) 0x7fefac138860 con 0x7fefac0b4d60 2024-03-25T19:42:44.648 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.648+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefac140da0 con 0x7fefac0b4d60 2024-03-25T19:42:44.648 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.648+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fef9c001b40 con 0x7fefac0b4d60 2024-03-25T19:42:44.648 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.648+0000 7fefa2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:44.648 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.648+0000 7fefa2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:44.648 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.648+0000 7fefa2ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:44.648 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:44.648 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:44.648 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:44.648 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:44.648 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:44.648 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:44.648 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:44.649 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:44.649 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:44.649 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:44.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.648+0000 7fefa2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:44.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.650+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2701102670 0 0) 0x7fef90004eb0 con 0x7fefac0b1730 2024-03-25T19:42:44.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.650+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefac138860 con 0x7fefac0b1730 2024-03-25T19:42:44.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.650+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fefa80018f0 con 0x7fefac0b1730 2024-03-25T19:42:44.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.650+0000 7fefa2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:44.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.650+0000 7fefa2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:44.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.650+0000 7fefa2ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:44.650 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:44.650 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:44.650 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:44.650 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:44.650 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:44.650 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:44.650 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:44.650 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:44.650 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:44.650 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:44.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.661+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2525853274 0 0) 0x7fefac141f10 con 0x7fefac14be90 2024-03-25T19:42:44.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.661+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef90004eb0 con 0x7fefac14be90 2024-03-25T19:42:44.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.661+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fefa4001bd0 con 0x7fefac14be90 2024-03-25T19:42:44.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.661+0000 7fefa2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:44.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.661+0000 7fefa2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:44.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.661+0000 7fefa2ffd640 10 monclient(hunting): dump: 2024-03-25T19:42:44.661 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:44.661 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:44.661 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:44.662 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:44.662 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:44.662 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:44.662 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:44.662 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:44.662 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:44.662 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3214890094 0 0) 0x7fefac138860 con 0x7fefac0b1730 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 >> v1:172.21.15.98:6789/0 conn(0x7fefac14be90 legacy=0x7fefac1e2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 >> v1:172.21.15.81:6789/0 conn(0x7fefac0b4d60 legacy=0x7fefac1d7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefa2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefa2ffd640 1 monclient: found mon.c 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefa2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefa2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:44.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac137f50 con 0x7fefac0b1730 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefa2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefa2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefa2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefb42a8640 5 monclient: authenticate success, global_id 4244 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefb42a8640 10 monclient: _renew_subs 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefb42a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefb42a8640 1 -- 172.21.15.81:0/3231988985 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefac1d8080 con 0x7fefac0b1730 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefb42a8640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefb42a8640 10 monclient: _renew_subs 2024-03-25T19:42:44.667 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefb42a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:44.668 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefb42a8640 1 -- 172.21.15.81:0/3231988985 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefac1d8510 con 0x7fefac0b1730 2024-03-25T19:42:44.668 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.666+0000 7fefb42a8640 1 librados: init done 2024-03-25T19:42:44.668 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.667+0000 7fefb42a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:44.668 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.667+0000 7fefb42a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:44.668 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.667+0000 7fefb42a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:44.668 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.667+0000 7fefb42a8640 1 -- 172.21.15.81:0/3231988985 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefac1381a0 con 0x7fefac0b1730 2024-03-25T19:42:44.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.996+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefa8003610 con 0x7fefac0b1730 2024-03-25T19:42:44.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.996+0000 7fefa2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:44.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.996+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fefa8003920 con 0x7fefac0b1730 2024-03-25T19:42:44.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.996+0000 7fefa2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:44.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.996+0000 7fefa2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:44.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.996+0000 7fefa2ffd640 10 monclient: dump: 2024-03-25T19:42:44.996 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:44.996 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:44.997 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:44.997 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:44.997 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:44.997 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:44.997 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:44.997 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:44.997 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:44.997 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:44.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.996+0000 7fefb0841640 4 set_mon_vals no callback set 2024-03-25T19:42:44.998 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.997+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fefa8019930 con 0x7fefac0b1730 2024-03-25T19:42:44.998 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.997+0000 7fefb1843640 10 monclient: get_auth_request con 0x7fef9006e160 auth_method 0 2024-03-25T19:42:44.998 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:44.998+0000 7fefa2ffd640 10 client.4244.objecter ms_handle_connect 0x7fef9006e160 2024-03-25T19:42:45.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.002+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fefa80535d0 con 0x7fefac0b1730 2024-03-25T19:42:45.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.002+0000 7fefa2ffd640 10 client.4244.objecter ms_dispatch 0x7fefac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:45.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.002+0000 7fefa2ffd640 3 client.4244.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:45.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.002+0000 7fefa2ffd640 3 client.4244.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:45.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.002+0000 7fefa2ffd640 20 client.4244.objecter dump_active .. 0 homeless 2024-03-25T19:42:45.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.002+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fefa804ef10 con 0x7fefac0b1730 2024-03-25T19:42:45.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.002+0000 7fefa2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:45.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.002+0000 7fefa2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:45.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.246+0000 7fefb42a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:45.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.246+0000 7fefb42a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:45.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.246+0000 7fefb42a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:45.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.246+0000 7fefb42a8640 1 -- 172.21.15.81:0/3231988985 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefac1d7690 con 0x7fefac0b1730 2024-03-25T19:42:45.325 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.326+0000 7fefa2ffd640 1 -- 172.21.15.81:0/3231988985 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fefa8051010 con 0x7fefac0b1730 2024-03-25T19:42:45.325 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.326+0000 7fefa2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:45.326 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.326+0000 7fefa2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:45.326 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:42:45.326 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:45.328 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.328+0000 7fefb42a8640 10 librados: watch_flush enter 2024-03-25T19:42:45.328 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.328+0000 7fefb42a8640 10 librados: watch_flush exit 2024-03-25T19:42:45.328 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.328+0000 7fefb42a8640 20 client.4244.objecter shutdown clearing up homeless session... 2024-03-25T19:42:45.328 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.328+0000 7fefb42a8640 10 client.4244.objecter successfully canceled tick 2024-03-25T19:42:45.328 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.328+0000 7fefb42a8640 1 -- 172.21.15.81:0/3231988985 >> v1:172.21.15.98:6800/3980829775 conn(0x7fef9006e160 legacy=0x7fef90070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:45.328 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.328+0000 7fefb42a8640 10 monclient: shutdown 2024-03-25T19:42:45.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.328+0000 7fefb42a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:45.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.328+0000 7fefb42a8640 1 -- 172.21.15.81:0/3231988985 >> v1:172.21.15.81:6790/0 conn(0x7fefac0b1730 legacy=0x7fefac1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:45.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.328+0000 7fefb42a8640 1 -- 172.21.15.81:0/3231988985 shutdown_connections 2024-03-25T19:42:45.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.328+0000 7fefb42a8640 1 -- 172.21.15.81:0/3231988985 >> 172.21.15.81:0/3231988985 conn(0x7fefac0617a0 msgr2=0x7fefac0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:45.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.328+0000 7fefb42a8640 1 -- 172.21.15.81:0/3231988985 shutdown_connections 2024-03-25T19:42:45.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.328+0000 7fefb42a8640 1 -- 172.21.15.81:0/3231988985 wait complete. 2024-03-25T19:42:45.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:45.328+0000 7fefb42a8640 1 librados: shutdown 2024-03-25T19:42:45.339 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:52.145 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:52.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.258+0000 7fd7b008d640 10 monclient: get_monmap_and_config 2024-03-25T19:42:52.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.258+0000 7fd7b008d640 10 monclient: build_initial_monmap 2024-03-25T19:42:52.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.259+0000 7fd7b008d640 10 monclient: monmap: 2024-03-25T19:42:52.259 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:42:52.259 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:52.259 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:42:52.259910+0000 2024-03-25T19:42:52.259 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:42:52.259910+0000 2024-03-25T19:42:52.259 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:52.259 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:52.259 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:52.259 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:42:52.260 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:42:52.260 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:52.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.259+0000 7fd7b008d640 1 Processor -- start 2024-03-25T19:42:52.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.260+0000 7fd7b008d640 1 -- start start 2024-03-25T19:42:52.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.260+0000 7fd7b008d640 10 monclient: init 2024-03-25T19:42:52.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.261+0000 7fd7b008d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:52.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.261+0000 7fd7b008d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:52.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.261+0000 7fd7b008d640 10 monclient(hunting): picked mon.noname-a con 0x7fd7a80a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:52.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.261+0000 7fd7b008d640 10 monclient(hunting): picked mon.noname-b con 0x7fd7a813f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:52.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.261+0000 7fd7b008d640 10 monclient(hunting): picked mon.noname-c con 0x7fd7a81430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:52.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.261+0000 7fd7b008d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a8142520 con 0x7fd7a80a98f0 2024-03-25T19:42:52.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.261+0000 7fd7b008d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a80b28a0 con 0x7fd7a81430c0 2024-03-25T19:42:52.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.261+0000 7fd7b008d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a80b2a30 con 0x7fd7a813f710 2024-03-25T19:42:52.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.261+0000 7fd7b008d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:52.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.261+0000 7fd7b008d640 10 monclient(hunting): authenticate will time out at 4007.343308s 2024-03-25T19:42:52.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.265+0000 7fd7ade29640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fd7a80a98f0 0x7fd7a80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:55358/0 (socket says 172.21.15.81:55358) 2024-03-25T19:42:52.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.265+0000 7fd7ade29640 1 -- 172.21.15.81:0/3963251096 learned_addr learned my addr 172.21.15.81:0/3963251096 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:52.276 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.276+0000 7fd7ace27640 1 -- 172.21.15.81:0/3963251096 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408141693 0 0) 0x7fd7a8142520 con 0x7fd7a80a98f0 2024-03-25T19:42:52.276 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.276+0000 7fd7ace27640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.276 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.276+0000 7fd7ace27640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.276 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.276+0000 7fd7ace27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.276 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.276+0000 7fd7ace27640 10 monclient(hunting): my global_id is 4260 2024-03-25T19:42:52.276 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.276+0000 7fd7ace27640 1 -- 172.21.15.81:0/3963251096 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd794003520 con 0x7fd7a80a98f0 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.282+0000 7fd7ace27640 1 -- 172.21.15.81:0/3963251096 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303769627 0 0) 0x7fd7a80b28a0 con 0x7fd7a81430c0 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.282+0000 7fd7ace27640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.282+0000 7fd7ace27640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.282+0000 7fd7ace27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.282+0000 7fd7ace27640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:42:52.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.282+0000 7fd7ace27640 1 -- 172.21.15.81:0/3963251096 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a8142520 con 0x7fd7a81430c0 2024-03-25T19:42:52.290 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7ace27640 1 -- 172.21.15.81:0/3963251096 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2872609716 0 0) 0x7fd794003520 con 0x7fd7a80a98f0 2024-03-25T19:42:52.290 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7ace27640 1 -- 172.21.15.81:0/3963251096 >> v1:172.21.15.81:6790/0 conn(0x7fd7a813f710 legacy=0x7fd7a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.290 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7ace27640 1 -- 172.21.15.81:0/3963251096 >> v1:172.21.15.98:6789/0 conn(0x7fd7a81430c0 legacy=0x7fd7a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.290 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7ace27640 10 monclient: _finish_hunting 0 2024-03-25T19:42:52.290 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7ace27640 1 monclient: found mon.noname-a 2024-03-25T19:42:52.290 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7ace27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:52.290 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7ace27640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:42:52.290 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7ace27640 1 -- 172.21.15.81:0/3963251096 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a80b44c0 con 0x7fd7a80a98f0 2024-03-25T19:42:52.291 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7ace27640 10 monclient: _finish_auth 0 2024-03-25T19:42:52.291 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7ace27640 10 monclient: _check_auth_tickets 2024-03-25T19:42:52.291 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7ace27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:52.291 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7ace27640 1 -- 172.21.15.81:0/3963251096 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd7a4001690 con 0x7fd7a80a98f0 2024-03-25T19:42:52.291 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7b008d640 5 monclient: authenticate success, global_id 4260 2024-03-25T19:42:52.291 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7b008d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:52.291 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7ace27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.291 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7ace27640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:52.291 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7ace27640 10 monclient: dump: 2024-03-25T19:42:52.291 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:52.291 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:52.292 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:52.292 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:52.292 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.292 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:52.292 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:52.292 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:52.292 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:52.292 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:52.292 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.290+0000 7fd7b008d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:52.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.308+0000 7fd7ace27640 1 -- 172.21.15.81:0/3963251096 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7a40017f0 con 0x7fd7a80a98f0 2024-03-25T19:42:52.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.308+0000 7fd7ace27640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:52.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.308+0000 7fd7ace27640 1 -- 172.21.15.81:0/3963251096 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd7a4001c00 con 0x7fd7a80a98f0 2024-03-25T19:42:52.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.308+0000 7fd7ace27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.308+0000 7fd7ace27640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:52.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.308+0000 7fd7ace27640 10 monclient: dump: 2024-03-25T19:42:52.308 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:52.308 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:52.309 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:52.309 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:52.309 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.309 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:52.309 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:52.309 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:52.309 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:52.309 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:52.309 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.308+0000 7fd7b008d640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:52.309 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.308+0000 7fd7b008d640 4 set_mon_vals no callback set 2024-03-25T19:42:52.312 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.309+0000 7fd7b008d640 10 monclient: shutdown 2024-03-25T19:42:52.312 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.309+0000 7fd7b008d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:52.312 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.309+0000 7fd7b008d640 1 -- 172.21.15.81:0/3963251096 >> v1:172.21.15.81:6789/0 conn(0x7fd7a80a98f0 legacy=0x7fd7a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.312 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.309+0000 7fd7b008d640 1 -- 172.21.15.81:0/3963251096 shutdown_connections 2024-03-25T19:42:52.312 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.309+0000 7fd7b008d640 1 -- 172.21.15.81:0/3963251096 >> 172.21.15.81:0/3963251096 conn(0x7fd7a80b7ea0 msgr2=0x7fd7a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:52.312 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.309+0000 7fd7b008d640 1 -- 172.21.15.81:0/3963251096 shutdown_connections 2024-03-25T19:42:52.312 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.309+0000 7fd7b008d640 1 -- 172.21.15.81:0/3963251096 wait complete. 2024-03-25T19:42:52.312 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.310+0000 7fd7b008d640 10 monclient: build_initial_monmap 2024-03-25T19:42:52.313 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.310+0000 7fd7b008d640 10 monclient: monmap: 2024-03-25T19:42:52.313 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:42:52.313 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:52.313 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:42:52.313 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:42:52.313 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:52.313 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:42:52.313 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:52.313 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:52.313 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:52.313 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:52.313 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.310+0000 7fd7b008d640 1 librados: starting msgr at 2024-03-25T19:42:52.313 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.310+0000 7fd7b008d640 1 librados: starting objecter 2024-03-25T19:42:52.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.310+0000 7fd7b008d640 1 Processor -- start 2024-03-25T19:42:52.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.310+0000 7fd7b008d640 1 -- start start 2024-03-25T19:42:52.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.310+0000 7fd7b008d640 1 librados: setting wanted keys 2024-03-25T19:42:52.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.310+0000 7fd7b008d640 1 librados: calling monclient init 2024-03-25T19:42:52.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.310+0000 7fd7b008d640 10 monclient: init 2024-03-25T19:42:52.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.311+0000 7fd7b008d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:52.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.311+0000 7fd7b008d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:52.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.311+0000 7fd7b008d640 10 monclient(hunting): picked mon.noname-b con 0x7fd7a80a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:52.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.311+0000 7fd7b008d640 10 monclient(hunting): picked mon.noname-c con 0x7fd7a813f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:52.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.311+0000 7fd7b008d640 10 monclient(hunting): picked mon.noname-a con 0x7fd7a81430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:52.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.311+0000 7fd7b008d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a81d7910 con 0x7fd7a81430c0 2024-03-25T19:42:52.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.311+0000 7fd7b008d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a80abfe0 con 0x7fd7a80a98f0 2024-03-25T19:42:52.315 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.311+0000 7fd7b008d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a81d84b0 con 0x7fd7a813f710 2024-03-25T19:42:52.315 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.311+0000 7fd7b008d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:52.315 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.311+0000 7fd7b008d640 10 monclient(hunting): authenticate will time out at 4007.393276s 2024-03-25T19:42:52.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.319+0000 7fd7ade29640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd7a80a98f0 0x7fd7a81d7da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:55822/0 (socket says 172.21.15.81:55822) 2024-03-25T19:42:52.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.320+0000 7fd7ade29640 1 -- 172.21.15.81:0/2289394949 learned_addr learned my addr 172.21.15.81:0/2289394949 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:52.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.321+0000 7fd79e7fc640 10 client.?.objecter ms_handle_connect 0x7fd7a80a98f0 2024-03-25T19:42:52.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.322+0000 7fd79e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:52.323 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.323+0000 7fd79e7fc640 10 client.?.objecter ms_handle_connect 0x7fd7a813f710 2024-03-25T19:42:52.323 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.323+0000 7fd79e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:52.327 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.327+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461328066 0 0) 0x7fd7a81d84b0 con 0x7fd7a813f710 2024-03-25T19:42:52.327 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.327+0000 7fd79e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.327 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.327+0000 7fd79e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.327 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.327+0000 7fd79e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.327 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.327+0000 7fd79e7fc640 10 monclient(hunting): my global_id is 4250 2024-03-25T19:42:52.327 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.327+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd78c004eb0 con 0x7fd7a813f710 2024-03-25T19:42:52.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.329+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4139924378 0 0) 0x7fd7a80abfe0 con 0x7fd7a80a98f0 2024-03-25T19:42:52.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.329+0000 7fd79e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.329+0000 7fd79e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.329+0000 7fd79e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.329+0000 7fd79e7fc640 10 monclient(hunting): my global_id is 4267 2024-03-25T19:42:52.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.329+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a81d84b0 con 0x7fd7a80a98f0 2024-03-25T19:42:52.331 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.332+0000 7fd79e7fc640 10 client.?.objecter ms_handle_connect 0x7fd7a81430c0 2024-03-25T19:42:52.331 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.332+0000 7fd79e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:52.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.337+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067555477 0 0) 0x7fd7a81d7910 con 0x7fd7a81430c0 2024-03-25T19:42:52.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.337+0000 7fd79e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:52.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.337+0000 7fd79e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:52.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.337+0000 7fd79e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:52.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.337+0000 7fd79e7fc640 10 monclient(hunting): my global_id is 4263 2024-03-25T19:42:52.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.337+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a80abfe0 con 0x7fd7a81430c0 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.341+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3778055614 0 0) 0x7fd78c004eb0 con 0x7fd7a813f710 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.341+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7a81d7910 con 0x7fd7a813f710 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.341+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd798001930 con 0x7fd7a813f710 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.341+0000 7fd79e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.341+0000 7fd79e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:52.340 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.341+0000 7fd79e7fc640 10 monclient(hunting): dump: 2024-03-25T19:42:52.341 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:52.341 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:52.341 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:52.341 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:52.341 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.341 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:52.341 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:52.341 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:52.341 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:52.341 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:52.341 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.341+0000 7fd79e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:52.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.342+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3891055894 0 0) 0x7fd7a81d84b0 con 0x7fd7a80a98f0 2024-03-25T19:42:52.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.342+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd78c004eb0 con 0x7fd7a80a98f0 2024-03-25T19:42:52.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.342+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd7a4001b40 con 0x7fd7a80a98f0 2024-03-25T19:42:52.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.342+0000 7fd79e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.342+0000 7fd79e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:52.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.342+0000 7fd79e7fc640 10 monclient(hunting): dump: 2024-03-25T19:42:52.342 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:52.342 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:52.342 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:52.343 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:52.343 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.343 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:52.343 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:52.343 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:52.343 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:52.343 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:52.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2100919377 0 0) 0x7fd7a81d7910 con 0x7fd7a813f710 2024-03-25T19:42:52.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 >> v1:172.21.15.98:6789/0 conn(0x7fd7a80a98f0 legacy=0x7fd7a81d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 >> v1:172.21.15.81:6789/0 conn(0x7fd7a81430c0 legacy=0x7fd7a8138d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd79e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:52.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd79e7fc640 1 monclient: found mon.c 2024-03-25T19:42:52.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd79e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:52.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd79e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:52.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a81d51f0 con 0x7fd7a813f710 2024-03-25T19:42:52.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd79e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:52.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd79e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:52.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd79e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:52.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd7b008d640 5 monclient: authenticate success, global_id 4250 2024-03-25T19:42:52.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd7b008d640 10 monclient: _renew_subs 2024-03-25T19:42:52.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd7b008d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:52.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd7b008d640 1 -- 172.21.15.81:0/2289394949 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a81e2ed0 con 0x7fd7a813f710 2024-03-25T19:42:52.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd7b008d640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:52.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd7b008d640 10 monclient: _renew_subs 2024-03-25T19:42:52.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd7b008d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:52.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd7b008d640 1 -- 172.21.15.81:0/2289394949 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a81e3240 con 0x7fd7a813f710 2024-03-25T19:42:52.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.349+0000 7fd7b008d640 1 librados: init done 2024-03-25T19:42:52.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.350+0000 7fd7b008d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:52.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.350+0000 7fd7b008d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:52.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.350+0000 7fd7b008d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:52.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.350+0000 7fd7b008d640 1 -- 172.21.15.81:0/2289394949 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a80627b0 con 0x7fd7a813f710 2024-03-25T19:42:52.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.644+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7980035f0 con 0x7fd7a813f710 2024-03-25T19:42:52.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.644+0000 7fd79e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:52.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.644+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd798003900 con 0x7fd7a813f710 2024-03-25T19:42:52.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.644+0000 7fd79e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:52.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.644+0000 7fd79e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:52.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.644+0000 7fd79e7fc640 10 monclient: dump: 2024-03-25T19:42:52.644 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:52.644 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:52.644 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:52.645 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:52.645 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:52.645 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:52.645 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:52.645 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:52.645 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:52.645 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:52.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.644+0000 7fd79ffff640 4 set_mon_vals no callback set 2024-03-25T19:42:52.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.645+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fd7980198d0 con 0x7fd7a813f710 2024-03-25T19:42:52.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.645+0000 7fd7ade29640 10 monclient: get_auth_request con 0x7fd78c06df20 auth_method 0 2024-03-25T19:42:52.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.645+0000 7fd79e7fc640 10 client.4250.objecter ms_handle_connect 0x7fd78c06df20 2024-03-25T19:42:52.651 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.651+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fd7980535e0 con 0x7fd7a813f710 2024-03-25T19:42:52.651 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.651+0000 7fd79e7fc640 10 client.4250.objecter ms_dispatch 0x7fd7a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:52.651 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.651+0000 7fd79e7fc640 3 client.4250.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:52.651 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.651+0000 7fd79e7fc640 3 client.4250.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:52.651 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.651+0000 7fd79e7fc640 20 client.4250.objecter dump_active .. 0 homeless 2024-03-25T19:42:52.651 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.651+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd79804f0f0 con 0x7fd7a813f710 2024-03-25T19:42:52.651 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.651+0000 7fd79e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:52.651 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.651+0000 7fd79e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:52.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.898+0000 7fd7b008d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:52.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.898+0000 7fd7b008d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:52.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.898+0000 7fd7b008d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:42:52.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.898+0000 7fd7b008d640 1 -- 172.21.15.81:0/2289394949 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7a813df50 con 0x7fd7a813f710 2024-03-25T19:42:52.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.940+0000 7fd79e7fc640 1 -- 172.21.15.81:0/2289394949 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fd798051020 con 0x7fd7a813f710 2024-03-25T19:42:52.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.940+0000 7fd79e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:52.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.940+0000 7fd79e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:52.940 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:42:52.940 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:52.942 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.942+0000 7fd7b008d640 10 librados: watch_flush enter 2024-03-25T19:42:52.943 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.942+0000 7fd7b008d640 10 librados: watch_flush exit 2024-03-25T19:42:52.943 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.942+0000 7fd7b008d640 20 client.4250.objecter shutdown clearing up homeless session... 2024-03-25T19:42:52.943 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.942+0000 7fd7b008d640 10 client.4250.objecter successfully canceled tick 2024-03-25T19:42:52.943 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.942+0000 7fd7b008d640 1 -- 172.21.15.81:0/2289394949 >> v1:172.21.15.98:6800/3980829775 conn(0x7fd78c06df20 legacy=0x7fd78c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.943 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.942+0000 7fd7b008d640 10 monclient: shutdown 2024-03-25T19:42:52.943 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.942+0000 7fd7b008d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:52.943 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.942+0000 7fd7b008d640 1 -- 172.21.15.81:0/2289394949 >> v1:172.21.15.81:6790/0 conn(0x7fd7a813f710 legacy=0x7fd7a81385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:52.943 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.942+0000 7fd7b008d640 1 -- 172.21.15.81:0/2289394949 shutdown_connections 2024-03-25T19:42:52.943 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.942+0000 7fd7b008d640 1 -- 172.21.15.81:0/2289394949 >> 172.21.15.81:0/2289394949 conn(0x7fd7a80b7ea0 msgr2=0x7fd7a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:52.943 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.942+0000 7fd7b008d640 1 -- 172.21.15.81:0/2289394949 shutdown_connections 2024-03-25T19:42:52.943 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.942+0000 7fd7b008d640 1 -- 172.21.15.81:0/2289394949 wait complete. 2024-03-25T19:42:52.943 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:52.943+0000 7fd7b008d640 1 librados: shutdown 2024-03-25T19:42:52.954 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:56.741 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:01:19.383 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:56.794 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:01:19.436 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:56.830 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:19.464 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:56.895 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:19.529 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:56.944 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:01:19.586 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:57.003 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:01:19.645 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:57.026 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:01:19.668 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:57.123 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:19.758 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:57.252 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:01:19.893 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:59.761 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:59.882 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.882+0000 7f8e10d3d640 10 monclient: get_monmap_and_config 2024-03-25T19:42:59.882 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.882+0000 7f8e10d3d640 10 monclient: build_initial_monmap 2024-03-25T19:42:59.882 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.882+0000 7f8e10d3d640 10 monclient: monmap: 2024-03-25T19:42:59.882 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:42:59.883 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:59.883 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:42:59.883853+0000 2024-03-25T19:42:59.883 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:42:59.883853+0000 2024-03-25T19:42:59.883 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:59.883 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:59.883 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:59.883 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:42:59.883 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:42:59.883 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:59.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.883+0000 7f8e10d3d640 1 Processor -- start 2024-03-25T19:42:59.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.883+0000 7f8e10d3d640 1 -- start start 2024-03-25T19:42:59.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.883+0000 7f8e10d3d640 10 monclient: init 2024-03-25T19:42:59.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.884+0000 7f8e10d3d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:59.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.884+0000 7f8e10d3d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:59.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.884+0000 7f8e10d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f8e0c0b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:59.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.884+0000 7f8e10d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f8e0c0b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:59.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.884+0000 7f8e10d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f8e0c14be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:59.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.884+0000 7f8e10d3d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0c0af940 con 0x7f8e0c0b4d60 2024-03-25T19:42:59.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.884+0000 7f8e10d3d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0c0b0a70 con 0x7f8e0c14be90 2024-03-25T19:42:59.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.884+0000 7f8e10d3d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0c0b0d50 con 0x7f8e0c0b1730 2024-03-25T19:42:59.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.884+0000 7f8e10d3d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:59.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.884+0000 7f8e10d3d640 10 monclient(hunting): authenticate will time out at 4014.966509s 2024-03-25T19:42:59.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.894+0000 7f8e0ad9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8e0c14be90 0x7f8e0c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:59054/0 (socket says 172.21.15.81:59054) 2024-03-25T19:42:59.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.894+0000 7f8e0ad9d640 1 -- 172.21.15.81:0/2227243518 learned_addr learned my addr 172.21.15.81:0/2227243518 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:59.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.917+0000 7f8e09d9b640 1 -- 172.21.15.81:0/2227243518 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195155476 0 0) 0x7f8e0c0b0a70 con 0x7f8e0c14be90 2024-03-25T19:42:59.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.917+0000 7f8e09d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:59.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.917+0000 7f8e09d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:59.917 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.917+0000 7f8e09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:59.917 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.917+0000 7f8e09d9b640 10 monclient(hunting): my global_id is 4270 2024-03-25T19:42:59.917 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.917+0000 7f8e09d9b640 1 -- 172.21.15.81:0/2227243518 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8df8003520 con 0x7f8e0c14be90 2024-03-25T19:42:59.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.928+0000 7f8e09d9b640 1 -- 172.21.15.81:0/2227243518 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275710617 0 0) 0x7f8e0c0af940 con 0x7f8e0c0b4d60 2024-03-25T19:42:59.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.928+0000 7f8e09d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:59.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.928+0000 7f8e09d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:59.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.928+0000 7f8e09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:59.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.928+0000 7f8e09d9b640 10 monclient(hunting): my global_id is 4266 2024-03-25T19:42:59.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.929+0000 7f8e09d9b640 1 -- 172.21.15.81:0/2227243518 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e0c0b0a70 con 0x7f8e0c0b4d60 2024-03-25T19:42:59.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.934+0000 7f8e09d9b640 1 -- 172.21.15.81:0/2227243518 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241486410 0 0) 0x7f8e0c0b0d50 con 0x7f8e0c0b1730 2024-03-25T19:42:59.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.934+0000 7f8e09d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:42:59.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.934+0000 7f8e09d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:59.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.934+0000 7f8e09d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:59.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.934+0000 7f8e09d9b640 10 monclient(hunting): my global_id is 4253 2024-03-25T19:42:59.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.934+0000 7f8e09d9b640 1 -- 172.21.15.81:0/2227243518 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e0c0af940 con 0x7f8e0c0b1730 2024-03-25T19:42:59.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.940+0000 7f8e09d9b640 1 -- 172.21.15.81:0/2227243518 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1179883962 0 0) 0x7f8df8003520 con 0x7f8e0c14be90 2024-03-25T19:42:59.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.940+0000 7f8e09d9b640 1 -- 172.21.15.81:0/2227243518 >> v1:172.21.15.81:6790/0 conn(0x7f8e0c0b1730 legacy=0x7f8e0c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:59.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.940+0000 7f8e09d9b640 1 -- 172.21.15.81:0/2227243518 >> v1:172.21.15.81:6789/0 conn(0x7f8e0c0b4d60 legacy=0x7f8e0c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:59.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.940+0000 7f8e09d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:42:59.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.940+0000 7f8e09d9b640 1 monclient: found mon.noname-c 2024-03-25T19:42:59.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.940+0000 7f8e09d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:59.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.940+0000 7f8e09d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:42:59.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.940+0000 7f8e09d9b640 1 -- 172.21.15.81:0/2227243518 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0c0b1030 con 0x7f8e0c14be90 2024-03-25T19:42:59.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.940+0000 7f8e09d9b640 10 monclient: _finish_auth 0 2024-03-25T19:42:59.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.940+0000 7f8e09d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:42:59.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.940+0000 7f8e09d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:59.941 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.940+0000 7f8e09d9b640 1 -- 172.21.15.81:0/2227243518 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8e04001a10 con 0x7f8e0c14be90 2024-03-25T19:42:59.941 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.940+0000 7f8e09d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:59.941 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.940+0000 7f8e09d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:59.941 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.940+0000 7f8e09d9b640 10 monclient: dump: 2024-03-25T19:42:59.941 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:59.941 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:59.941 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:59.941 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:59.941 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:59.941 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:59.941 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:59.941 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:59.941 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:59.941 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:59.941 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.940+0000 7f8e10d3d640 5 monclient: authenticate success, global_id 4270 2024-03-25T19:42:59.942 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.940+0000 7f8e10d3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:59.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.965+0000 7f8e09d9b640 1 -- 172.21.15.81:0/2227243518 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e04001b70 con 0x7f8e0c14be90 2024-03-25T19:42:59.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.965+0000 7f8e09d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:59.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.965+0000 7f8e09d9b640 1 -- 172.21.15.81:0/2227243518 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8e04001f80 con 0x7f8e0c14be90 2024-03-25T19:42:59.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.965+0000 7f8e09d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:59.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.965+0000 7f8e09d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:59.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.965+0000 7f8e09d9b640 10 monclient: dump: 2024-03-25T19:42:59.965 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:42:59.965 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:42:59.965 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:59.965 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:42:59.965 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:59.965 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:42:59.966 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:42:59.966 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:42:59.966 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:42:59.966 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:59.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.965+0000 7f8e10d3d640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:59.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.965+0000 7f8e10d3d640 4 set_mon_vals no callback set 2024-03-25T19:42:59.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.966+0000 7f8e10d3d640 10 monclient: shutdown 2024-03-25T19:42:59.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.966+0000 7f8e10d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:59.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.966+0000 7f8e10d3d640 1 -- 172.21.15.81:0/2227243518 >> v1:172.21.15.98:6789/0 conn(0x7f8e0c14be90 legacy=0x7f8e0c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:59.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.966+0000 7f8e10d3d640 1 -- 172.21.15.81:0/2227243518 shutdown_connections 2024-03-25T19:42:59.968 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.966+0000 7f8e10d3d640 1 -- 172.21.15.81:0/2227243518 >> 172.21.15.81:0/2227243518 conn(0x7f8e0c0617a0 msgr2=0x7f8e0c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:59.968 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.966+0000 7f8e10d3d640 1 -- 172.21.15.81:0/2227243518 shutdown_connections 2024-03-25T19:42:59.968 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.966+0000 7f8e10d3d640 1 -- 172.21.15.81:0/2227243518 wait complete. 2024-03-25T19:42:59.968 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.967+0000 7f8e10d3d640 10 monclient: build_initial_monmap 2024-03-25T19:42:59.968 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.967+0000 7f8e10d3d640 10 monclient: monmap: 2024-03-25T19:42:59.968 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:42:59.968 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:59.968 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:42:59.968 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:42:59.968 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:59.968 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:42:59.968 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:42:59.968 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:42:59.968 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:42:59.969 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:42:59.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.967+0000 7f8e10d3d640 1 librados: starting msgr at 2024-03-25T19:42:59.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.967+0000 7f8e10d3d640 1 librados: starting objecter 2024-03-25T19:42:59.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.968+0000 7f8e10d3d640 1 Processor -- start 2024-03-25T19:42:59.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.968+0000 7f8e10d3d640 1 -- start start 2024-03-25T19:42:59.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.968+0000 7f8e10d3d640 1 librados: setting wanted keys 2024-03-25T19:42:59.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.968+0000 7f8e10d3d640 1 librados: calling monclient init 2024-03-25T19:42:59.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.968+0000 7f8e10d3d640 10 monclient: init 2024-03-25T19:42:59.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.968+0000 7f8e10d3d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:59.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.968+0000 7f8e10d3d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:59.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.968+0000 7f8e10d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f8e0c0b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:42:59.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.968+0000 7f8e10d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f8e0c0b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:42:59.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.968+0000 7f8e10d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f8e0c14be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:42:59.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.968+0000 7f8e10d3d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0c135ee0 con 0x7f8e0c0b1730 2024-03-25T19:42:59.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.968+0000 7f8e10d3d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0c135b00 con 0x7f8e0c14be90 2024-03-25T19:42:59.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.968+0000 7f8e10d3d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0c135c90 con 0x7f8e0c0b4d60 2024-03-25T19:42:59.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.968+0000 7f8e10d3d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:59.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.969+0000 7f8e10d3d640 10 monclient(hunting): authenticate will time out at 4015.050834s 2024-03-25T19:42:59.982 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.983+0000 7f8e0ad9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f8e0c14be90 0x7f8e0c1d7e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:59124/0 (socket says 172.21.15.81:59124) 2024-03-25T19:42:59.982 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.983+0000 7f8e0ad9d640 1 -- 172.21.15.81:0/1484146258 learned_addr learned my addr 172.21.15.81:0/1484146258 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:42:59.984 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.984+0000 7f8e02ffd640 10 client.?.objecter ms_handle_connect 0x7f8e0c14be90 2024-03-25T19:42:59.984 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.984+0000 7f8e02ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:59.992 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.992+0000 7f8e02ffd640 10 client.?.objecter ms_handle_connect 0x7f8e0c0b4d60 2024-03-25T19:42:59.992 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.992+0000 7f8e02ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:59.995 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.996+0000 7f8e02ffd640 10 client.?.objecter ms_handle_connect 0x7f8e0c0b1730 2024-03-25T19:42:59.995 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.996+0000 7f8e02ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:59.999 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.999+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482575109 0 0) 0x7f8e0c135b00 con 0x7f8e0c14be90 2024-03-25T19:42:59.999 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.999+0000 7f8e02ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:42:59.999 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.999+0000 7f8e02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:59.999 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.999+0000 7f8e02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:59.999 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.999+0000 7f8e02ffd640 10 monclient(hunting): my global_id is 4273 2024-03-25T19:42:59.999 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:42:59.999+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dec004eb0 con 0x7f8e0c14be90 2024-03-25T19:43:00.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.002+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484914425 0 0) 0x7f8e0c135c90 con 0x7f8e0c0b4d60 2024-03-25T19:43:00.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.002+0000 7f8e02ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:00.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.002+0000 7f8e02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:00.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.002+0000 7f8e02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:00.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.002+0000 7f8e02ffd640 10 monclient(hunting): my global_id is 4256 2024-03-25T19:43:00.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.003+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e0c135b00 con 0x7f8e0c0b4d60 2024-03-25T19:43:00.004 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.005+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430659227 0 0) 0x7f8e0c135ee0 con 0x7f8e0c0b1730 2024-03-25T19:43:00.004 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.005+0000 7f8e02ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:00.004 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.005+0000 7f8e02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:00.004 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.005+0000 7f8e02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:00.004 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.005+0000 7f8e02ffd640 10 monclient(hunting): my global_id is 4269 2024-03-25T19:43:00.004 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.005+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e0c135c90 con 0x7f8e0c0b1730 2024-03-25T19:43:00.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.012+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 741210085 0 0) 0x7f8e0c135b00 con 0x7f8e0c0b4d60 2024-03-25T19:43:00.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.012+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e0c135ee0 con 0x7f8e0c0b4d60 2024-03-25T19:43:00.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.012+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8df4001870 con 0x7f8e0c0b4d60 2024-03-25T19:43:00.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.012+0000 7f8e02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:00.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.012+0000 7f8e02ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:00.012 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.012+0000 7f8e02ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:00.012 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:00.012 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:00.012 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:00.012 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:00.012 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:00.012 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:00.012 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:00.012 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:00.012 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:00.012 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:00.012 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.012+0000 7f8e02ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:00.021 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.021+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2842904646 0 0) 0x7f8e0c135c90 con 0x7f8e0c0b1730 2024-03-25T19:43:00.021 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.021+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e0c135b00 con 0x7f8e0c0b1730 2024-03-25T19:43:00.021 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.021+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8dfc0018f0 con 0x7f8e0c0b1730 2024-03-25T19:43:00.021 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.021+0000 7f8e02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:00.021 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.021+0000 7f8e02ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:00.021 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.021+0000 7f8e02ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:00.021 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:00.021 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:00.021 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:00.021 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:00.022 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:00.022 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:00.022 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:00.022 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:00.022 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:00.022 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:00.032 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.032+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 765102142 0 0) 0x7f8dec004eb0 con 0x7f8e0c14be90 2024-03-25T19:43:00.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.032+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e0c135c90 con 0x7f8e0c14be90 2024-03-25T19:43:00.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.032+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8e04001ec0 con 0x7f8e0c14be90 2024-03-25T19:43:00.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.032+0000 7f8e02ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:00.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.032+0000 7f8e02ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:00.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.032+0000 7f8e02ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:00.033 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:00.033 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:00.033 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:00.033 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:00.033 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:00.033 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:00.033 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:00.034 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:00.034 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:00.034 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:00.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 462305670 0 0) 0x7f8e0c135b00 con 0x7f8e0c0b1730 2024-03-25T19:43:00.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 >> v1:172.21.15.81:6790/0 conn(0x7f8e0c0b4d60 legacy=0x7f8e0c1d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 >> v1:172.21.15.98:6789/0 conn(0x7f8e0c14be90 legacy=0x7f8e0c1d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e02ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:00.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e02ffd640 1 monclient: found mon.a 2024-03-25T19:43:00.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:00.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e02ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:43:00.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0c136960 con 0x7f8e0c0b1730 2024-03-25T19:43:00.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e02ffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:00.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e02ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:00.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:00.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e10d3d640 5 monclient: authenticate success, global_id 4269 2024-03-25T19:43:00.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e10d3d640 10 monclient: _renew_subs 2024-03-25T19:43:00.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e10d3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:43:00.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e10d3d640 1 -- 172.21.15.81:0/1484146258 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e0c130470 con 0x7f8e0c0b1730 2024-03-25T19:43:00.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e10d3d640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:00.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e10d3d640 10 monclient: _renew_subs 2024-03-25T19:43:00.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e10d3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:43:00.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e10d3d640 1 -- 172.21.15.81:0/1484146258 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e0c13e870 con 0x7f8e0c0b1730 2024-03-25T19:43:00.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.033+0000 7f8e10d3d640 1 librados: init done 2024-03-25T19:43:00.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.034+0000 7f8e10d3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:00.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.034+0000 7f8e10d3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:00.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.034+0000 7f8e10d3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:43:00.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.034+0000 7f8e10d3d640 1 -- 172.21.15.81:0/1484146258 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e0c130810 con 0x7f8e0c0b1730 2024-03-25T19:43:00.341 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.342+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8dfc0035f0 con 0x7f8e0c0b1730 2024-03-25T19:43:00.341 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.342+0000 7f8e02ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:00.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.342+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8dfc003760 con 0x7f8e0c0b1730 2024-03-25T19:43:00.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.342+0000 7f8e02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:00.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.342+0000 7f8e02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:00.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.342+0000 7f8e02ffd640 10 monclient: dump: 2024-03-25T19:43:00.342 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:00.342 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:00.342 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:00.342 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:00.342 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:00.342 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:00.342 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:00.343 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:00.343 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:00.343 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:00.343 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.342+0000 7f8e0959a640 4 set_mon_vals no callback set 2024-03-25T19:43:00.343 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.342+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f8dfc019730 con 0x7f8e0c0b1730 2024-03-25T19:43:00.343 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.343+0000 7f8e03fff640 10 monclient: get_auth_request con 0x7f8dec06e160 auth_method 0 2024-03-25T19:43:00.344 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.343+0000 7f8e02ffd640 10 client.4269.objecter ms_handle_connect 0x7f8dec06e160 2024-03-25T19:43:00.348 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.349+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f8dfc053390 con 0x7f8e0c0b1730 2024-03-25T19:43:00.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.349+0000 7f8e02ffd640 10 client.4269.objecter ms_dispatch 0x7f8e0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:00.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.349+0000 7f8e02ffd640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:00.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.349+0000 7f8e02ffd640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:00.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.349+0000 7f8e02ffd640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-25T19:43:00.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.349+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8dfc04eea0 con 0x7f8e0c0b1730 2024-03-25T19:43:00.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.349+0000 7f8e02ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:00.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.349+0000 7f8e02ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:00.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.567+0000 7f8e10d3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:00.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.567+0000 7f8e10d3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:00.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.567+0000 7f8e10d3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:43:00.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.567+0000 7f8e10d3d640 1 -- 172.21.15.81:0/1484146258 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e0c136af0 con 0x7f8e0c0b1730 2024-03-25T19:43:00.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.596+0000 7f8e02ffd640 1 -- 172.21.15.81:0/1484146258 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f8dfc050dd0 con 0x7f8e0c0b1730 2024-03-25T19:43:00.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.596+0000 7f8e02ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:00.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.596+0000 7f8e02ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:00.596 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:43:00.596 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:00.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.599+0000 7f8e10d3d640 10 librados: watch_flush enter 2024-03-25T19:43:00.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.599+0000 7f8e10d3d640 10 librados: watch_flush exit 2024-03-25T19:43:00.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.599+0000 7f8e10d3d640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-25T19:43:00.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.599+0000 7f8e10d3d640 10 client.4269.objecter successfully canceled tick 2024-03-25T19:43:00.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.599+0000 7f8e10d3d640 1 -- 172.21.15.81:0/1484146258 >> v1:172.21.15.98:6800/3980829775 conn(0x7f8dec06e160 legacy=0x7f8dec070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.599+0000 7f8e10d3d640 10 monclient: shutdown 2024-03-25T19:43:00.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.599+0000 7f8e10d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:00.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.599+0000 7f8e10d3d640 1 -- 172.21.15.81:0/1484146258 >> v1:172.21.15.81:6789/0 conn(0x7f8e0c0b1730 legacy=0x7f8e0c12fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:00.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.599+0000 7f8e10d3d640 1 -- 172.21.15.81:0/1484146258 shutdown_connections 2024-03-25T19:43:00.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.599+0000 7f8e10d3d640 1 -- 172.21.15.81:0/1484146258 >> 172.21.15.81:0/1484146258 conn(0x7f8e0c0617a0 msgr2=0x7f8e0c0b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:00.600 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.599+0000 7f8e10d3d640 1 -- 172.21.15.81:0/1484146258 shutdown_connections 2024-03-25T19:43:00.600 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.599+0000 7f8e10d3d640 1 -- 172.21.15.81:0/1484146258 wait complete. 2024-03-25T19:43:00.600 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:00.599+0000 7f8e10d3d640 1 librados: shutdown 2024-03-25T19:43:00.612 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:07.418 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:07.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.513+0000 7f7608e06640 10 monclient: get_monmap_and_config 2024-03-25T19:43:07.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.513+0000 7f7608e06640 10 monclient: build_initial_monmap 2024-03-25T19:43:07.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.513+0000 7f7608e06640 10 monclient: monmap: 2024-03-25T19:43:07.512 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:43:07.512 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:07.512 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:43:07.513933+0000 2024-03-25T19:43:07.513 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:43:07.513933+0000 2024-03-25T19:43:07.513 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:07.513 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:07.513 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:43:07.513 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:43:07.513 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:43:07.513 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:07.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.513+0000 7f7608e06640 1 Processor -- start 2024-03-25T19:43:07.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.513+0000 7f7608e06640 1 -- start start 2024-03-25T19:43:07.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.513+0000 7f7608e06640 10 monclient: init 2024-03-25T19:43:07.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.514+0000 7f7608e06640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:07.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.514+0000 7f7608e06640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:07.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.514+0000 7f7608e06640 10 monclient(hunting): picked mon.noname-b con 0x7f76040a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:43:07.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.514+0000 7f7608e06640 10 monclient(hunting): picked mon.noname-a con 0x7f760413f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:43:07.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.514+0000 7f7608e06640 10 monclient(hunting): picked mon.noname-c con 0x7f76041430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:43:07.515 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.514+0000 7f7608e06640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7604142520 con 0x7f760413f710 2024-03-25T19:43:07.515 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.514+0000 7f7608e06640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76040b28a0 con 0x7f76041430c0 2024-03-25T19:43:07.515 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.514+0000 7f7608e06640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76040b2a30 con 0x7f76040a98f0 2024-03-25T19:43:07.515 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.514+0000 7f7608e06640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:07.515 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.514+0000 7f7608e06640 10 monclient(hunting): authenticate will time out at 4022.596660s 2024-03-25T19:43:07.522 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.522+0000 7f7601d9b640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f760413f710 0x7f76041429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:50700/0 (socket says 172.21.15.81:50700) 2024-03-25T19:43:07.522 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.523+0000 7f7601d9b640 1 -- 172.21.15.81:0/3936259543 learned_addr learned my addr 172.21.15.81:0/3936259543 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:43:07.533 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.534+0000 7f760159a640 1 -- 172.21.15.81:0/3936259543 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295258113 0 0) 0x7f7604142520 con 0x7f760413f710 2024-03-25T19:43:07.534 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.534+0000 7f760159a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.534 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.534+0000 7f760159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.534 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.534+0000 7f760159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.534 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.534+0000 7f760159a640 10 monclient(hunting): my global_id is 4272 2024-03-25T19:43:07.534 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.534+0000 7f760159a640 1 -- 172.21.15.81:0/3936259543 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75ec003520 con 0x7f760413f710 2024-03-25T19:43:07.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f760159a640 1 -- 172.21.15.81:0/3936259543 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2832550925 0 0) 0x7f75ec003520 con 0x7f760413f710 2024-03-25T19:43:07.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f760159a640 1 -- 172.21.15.81:0/3936259543 >> v1:172.21.15.81:6790/0 conn(0x7f76040a98f0 legacy=0x7f76040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:43:07.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f760159a640 1 -- 172.21.15.81:0/3936259543 >> v1:172.21.15.98:6789/0 conn(0x7f76041430c0 legacy=0x7f760414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:43:07.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f760159a640 10 monclient: _finish_hunting 0 2024-03-25T19:43:07.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f760159a640 1 monclient: found mon.noname-a 2024-03-25T19:43:07.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f760159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:07.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f760159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:43:07.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f760159a640 1 -- 172.21.15.81:0/3936259543 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76040b44c0 con 0x7f760413f710 2024-03-25T19:43:07.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f760159a640 10 monclient: _finish_auth 0 2024-03-25T19:43:07.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f760159a640 10 monclient: _check_auth_tickets 2024-03-25T19:43:07.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f760159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:07.551 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f760159a640 1 -- 172.21.15.81:0/3936259543 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f75f8001690 con 0x7f760413f710 2024-03-25T19:43:07.551 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f7608e06640 5 monclient: authenticate success, global_id 4272 2024-03-25T19:43:07.551 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f7608e06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:07.551 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f760159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.551 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f760159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:07.551 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f760159a640 10 monclient: dump: 2024-03-25T19:43:07.551 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:07.551 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:07.551 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:07.551 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:07.551 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.551 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:07.551 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:07.551 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:07.551 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:07.551 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:07.552 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.550+0000 7f7608e06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:07.553 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.553+0000 7f760159a640 1 -- 172.21.15.81:0/3936259543 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75f8001860 con 0x7f760413f710 2024-03-25T19:43:07.553 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.553+0000 7f760159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:07.553 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.553+0000 7f760159a640 1 -- 172.21.15.81:0/3936259543 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f75f8001cb0 con 0x7f760413f710 2024-03-25T19:43:07.553 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.553+0000 7f7608e06640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:07.553 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.553+0000 7f7608e06640 4 set_mon_vals no callback set 2024-03-25T19:43:07.553 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.553+0000 7f760159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.553 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.553+0000 7f760159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:07.553 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.553+0000 7f760159a640 10 monclient: dump: 2024-03-25T19:43:07.553 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:07.553 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:07.553 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:07.553 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:07.554 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.554 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:07.554 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:07.554 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:07.554 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:07.554 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:07.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.554+0000 7f7608e06640 10 monclient: shutdown 2024-03-25T19:43:07.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.554+0000 7f7608e06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:07.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.554+0000 7f7608e06640 1 -- 172.21.15.81:0/3936259543 >> v1:172.21.15.81:6789/0 conn(0x7f760413f710 legacy=0x7f76041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.554+0000 7f7608e06640 1 -- 172.21.15.81:0/3936259543 shutdown_connections 2024-03-25T19:43:07.556 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.554+0000 7f7608e06640 1 -- 172.21.15.81:0/3936259543 >> 172.21.15.81:0/3936259543 conn(0x7f76040b7ea0 msgr2=0x7f76040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:07.556 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.554+0000 7f7608e06640 1 -- 172.21.15.81:0/3936259543 shutdown_connections 2024-03-25T19:43:07.556 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.554+0000 7f7608e06640 1 -- 172.21.15.81:0/3936259543 wait complete. 2024-03-25T19:43:07.556 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.555+0000 7f7608e06640 10 monclient: build_initial_monmap 2024-03-25T19:43:07.556 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.555+0000 7f7608e06640 10 monclient: monmap: 2024-03-25T19:43:07.556 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:43:07.556 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:07.556 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:43:07.556 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:43:07.556 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:07.556 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:43:07.556 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:43:07.556 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:43:07.556 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:43:07.557 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:07.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.555+0000 7f7608e06640 1 librados: starting msgr at 2024-03-25T19:43:07.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.555+0000 7f7608e06640 1 librados: starting objecter 2024-03-25T19:43:07.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.556+0000 7f7608e06640 1 Processor -- start 2024-03-25T19:43:07.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.556+0000 7f7608e06640 1 -- start start 2024-03-25T19:43:07.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.556+0000 7f7608e06640 1 librados: setting wanted keys 2024-03-25T19:43:07.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.556+0000 7f7608e06640 1 librados: calling monclient init 2024-03-25T19:43:07.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.556+0000 7f7608e06640 10 monclient: init 2024-03-25T19:43:07.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.556+0000 7f7608e06640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:07.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.556+0000 7f7608e06640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:07.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.556+0000 7f7608e06640 10 monclient(hunting): picked mon.noname-a con 0x7f76040a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:43:07.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.556+0000 7f7608e06640 10 monclient(hunting): picked mon.noname-c con 0x7f760413f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:43:07.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.556+0000 7f7608e06640 10 monclient(hunting): picked mon.noname-b con 0x7f76041430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:43:07.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.556+0000 7f7608e06640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76040b07c0 con 0x7f76040a98f0 2024-03-25T19:43:07.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.556+0000 7f7608e06640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76040aef00 con 0x7f76041430c0 2024-03-25T19:43:07.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.556+0000 7f7608e06640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76040b0a30 con 0x7f760413f710 2024-03-25T19:43:07.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.556+0000 7f7608e06640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:07.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.556+0000 7f7608e06640 10 monclient(hunting): authenticate will time out at 4022.638679s 2024-03-25T19:43:07.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.565+0000 7f7601d9b640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f760413f710 0x7f76040b11c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:57516/0 (socket says 172.21.15.81:57516) 2024-03-25T19:43:07.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.565+0000 7f7601d9b640 1 -- 172.21.15.81:0/2452649876 learned_addr learned my addr 172.21.15.81:0/2452649876 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:43:07.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.567+0000 7f75f2ffd640 10 client.?.objecter ms_handle_connect 0x7f760413f710 2024-03-25T19:43:07.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.567+0000 7f75f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:07.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.569+0000 7f75f2ffd640 10 client.?.objecter ms_handle_connect 0x7f76041430c0 2024-03-25T19:43:07.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.569+0000 7f75f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:07.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.583+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101497951 0 0) 0x7f76040b0a30 con 0x7f760413f710 2024-03-25T19:43:07.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.583+0000 7f75f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.583+0000 7f75f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.583+0000 7f75f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.583+0000 7f75f2ffd640 10 monclient(hunting): my global_id is 4259 2024-03-25T19:43:07.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.583+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75e4004eb0 con 0x7f760413f710 2024-03-25T19:43:07.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.583+0000 7f75f2ffd640 10 client.?.objecter ms_handle_connect 0x7f76040a98f0 2024-03-25T19:43:07.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.583+0000 7f75f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:07.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.591+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66405616 0 0) 0x7f76040aef00 con 0x7f76041430c0 2024-03-25T19:43:07.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.591+0000 7f75f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.591+0000 7f75f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.591+0000 7f75f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.591+0000 7f75f2ffd640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:43:07.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.591+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76040b0a30 con 0x7f76041430c0 2024-03-25T19:43:07.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.605+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205945026 0 0) 0x7f76040b07c0 con 0x7f76040a98f0 2024-03-25T19:43:07.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.605+0000 7f75f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:07.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.605+0000 7f75f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:07.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.605+0000 7f75f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:07.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.605+0000 7f75f2ffd640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:43:07.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.605+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76040aef00 con 0x7f76040a98f0 2024-03-25T19:43:07.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.606+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 509109657 0 0) 0x7f75e4004eb0 con 0x7f760413f710 2024-03-25T19:43:07.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.606+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76040b07c0 con 0x7f760413f710 2024-03-25T19:43:07.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.606+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f75f8001b70 con 0x7f760413f710 2024-03-25T19:43:07.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.606+0000 7f75f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.606+0000 7f75f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:07.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.606+0000 7f75f2ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:07.606 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:07.606 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:07.606 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:07.607 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:07.607 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.607 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:07.607 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:07.607 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:07.607 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:07.607 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:07.607 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.606+0000 7f75f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:07.614 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.614+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1484209297 0 0) 0x7f76040b0a30 con 0x7f76041430c0 2024-03-25T19:43:07.614 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.614+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75e4004eb0 con 0x7f76041430c0 2024-03-25T19:43:07.614 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.614+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f75fc001a70 con 0x7f76041430c0 2024-03-25T19:43:07.614 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.614+0000 7f75f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.614 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.614+0000 7f75f2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:07.614 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.614+0000 7f75f2ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:07.614 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:07.614 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:07.614 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:07.614 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:07.614 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.614 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:07.614 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:07.615 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:07.615 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:07.615 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:07.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.625+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1204553641 0 0) 0x7f76040aef00 con 0x7f76040a98f0 2024-03-25T19:43:07.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.625+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76040b0a30 con 0x7f76040a98f0 2024-03-25T19:43:07.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.625+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f75f40017d0 con 0x7f76040a98f0 2024-03-25T19:43:07.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.625+0000 7f75f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.625+0000 7f75f2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:07.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.625+0000 7f75f2ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:07.625 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:07.625 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:07.625 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:07.625 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:07.625 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.625 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:07.625 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:07.625 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:07.625 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:07.625 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:07.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 93259 0 0) 0x7f76040b07c0 con 0x7f760413f710 2024-03-25T19:43:07.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 >> v1:172.21.15.98:6789/0 conn(0x7f76041430c0 legacy=0x7f76040af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 >> v1:172.21.15.81:6789/0 conn(0x7f76040a98f0 legacy=0x7f76040abb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f75f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:07.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f75f2ffd640 1 monclient: found mon.c 2024-03-25T19:43:07.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f75f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:07.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f75f2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:43:07.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76040ae880 con 0x7f760413f710 2024-03-25T19:43:07.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f75f2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:07.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f75f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:07.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f75f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:07.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f7608e06640 5 monclient: authenticate success, global_id 4259 2024-03-25T19:43:07.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f7608e06640 10 monclient: _renew_subs 2024-03-25T19:43:07.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f7608e06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:43:07.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f7608e06640 1 -- 172.21.15.81:0/2452649876 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76040aeb30 con 0x7f760413f710 2024-03-25T19:43:07.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f7608e06640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:07.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f7608e06640 10 monclient: _renew_subs 2024-03-25T19:43:07.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f7608e06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:43:07.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f7608e06640 1 -- 172.21.15.81:0/2452649876 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76041e2e40 con 0x7f760413f710 2024-03-25T19:43:07.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.626+0000 7f7608e06640 1 librados: init done 2024-03-25T19:43:07.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.627+0000 7f7608e06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:07.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.627+0000 7f7608e06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:07.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.627+0000 7f7608e06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:43:07.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.627+0000 7f7608e06640 1 -- 172.21.15.81:0/2452649876 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76040af090 con 0x7f760413f710 2024-03-25T19:43:07.631 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.632+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75f8004bf0 con 0x7f760413f710 2024-03-25T19:43:07.632 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.632+0000 7f75f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:07.632 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.632+0000 7f7600d99640 4 set_mon_vals no callback set 2024-03-25T19:43:07.930 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.931+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f75f8004d60 con 0x7f760413f710 2024-03-25T19:43:07.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.931+0000 7f75f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:07.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.931+0000 7f75f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:07.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.931+0000 7f75f2ffd640 10 monclient: dump: 2024-03-25T19:43:07.931 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:07.931 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:07.931 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:07.931 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:07.931 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:07.932 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:07.932 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:07.932 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:07.932 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:07.932 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:07.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.931+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f75f801ad30 con 0x7f760413f710 2024-03-25T19:43:07.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.932+0000 7f760259c640 10 monclient: get_auth_request con 0x7f75e406e0b0 auth_method 0 2024-03-25T19:43:07.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.932+0000 7f75f2ffd640 10 client.4259.objecter ms_handle_connect 0x7f75e406e0b0 2024-03-25T19:43:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.939+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f75f8054880 con 0x7f760413f710 2024-03-25T19:43:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.939+0000 7f75f2ffd640 10 client.4259.objecter ms_dispatch 0x7f7604000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.939+0000 7f75f2ffd640 3 client.4259.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.939+0000 7f75f2ffd640 3 client.4259.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.939+0000 7f75f2ffd640 20 client.4259.objecter dump_active .. 0 homeless 2024-03-25T19:43:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.939+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75f8050390 con 0x7f760413f710 2024-03-25T19:43:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.939+0000 7f75f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:07.939+0000 7f75f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:08.189 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.190+0000 7f7608e06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:08.189 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.190+0000 7f7608e06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:08.189 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.190+0000 7f7608e06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:43:08.189 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.190+0000 7f7608e06640 1 -- 172.21.15.81:0/2452649876 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76040627b0 con 0x7f760413f710 2024-03-25T19:43:08.217 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.217+0000 7f75f2ffd640 1 -- 172.21.15.81:0/2452649876 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f75f80522c0 con 0x7f760413f710 2024-03-25T19:43:08.217 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.217+0000 7f75f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:08.217 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.217+0000 7f75f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:08.217 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:43:08.217 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:08.219 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.219+0000 7f7608e06640 10 librados: watch_flush enter 2024-03-25T19:43:08.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.219+0000 7f7608e06640 10 librados: watch_flush exit 2024-03-25T19:43:08.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.219+0000 7f7608e06640 20 client.4259.objecter shutdown clearing up homeless session... 2024-03-25T19:43:08.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.219+0000 7f7608e06640 10 client.4259.objecter successfully canceled tick 2024-03-25T19:43:08.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.219+0000 7f7608e06640 1 -- 172.21.15.81:0/2452649876 >> v1:172.21.15.98:6800/3980829775 conn(0x7f75e406e0b0 legacy=0x7f75e4070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:08.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.219+0000 7f7608e06640 10 monclient: shutdown 2024-03-25T19:43:08.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.219+0000 7f7608e06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:08.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.219+0000 7f7608e06640 1 -- 172.21.15.81:0/2452649876 >> v1:172.21.15.81:6790/0 conn(0x7f760413f710 legacy=0x7f76040b11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:08.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.220+0000 7f7608e06640 1 -- 172.21.15.81:0/2452649876 shutdown_connections 2024-03-25T19:43:08.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.220+0000 7f7608e06640 1 -- 172.21.15.81:0/2452649876 >> 172.21.15.81:0/2452649876 conn(0x7f76040b7ea0 msgr2=0x7f7604074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:08.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.220+0000 7f7608e06640 1 -- 172.21.15.81:0/2452649876 shutdown_connections 2024-03-25T19:43:08.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.220+0000 7f7608e06640 1 -- 172.21.15.81:0/2452649876 wait complete. 2024-03-25T19:43:08.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:08.220+0000 7f7608e06640 1 librados: shutdown 2024-03-25T19:43:08.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:15.038 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:15.158 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.159+0000 7fa8c2332640 10 monclient: get_monmap_and_config 2024-03-25T19:43:15.158 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.159+0000 7fa8c2332640 10 monclient: build_initial_monmap 2024-03-25T19:43:15.158 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.159+0000 7fa8c2332640 10 monclient: monmap: 2024-03-25T19:43:15.159 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:43:15.159 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:15.159 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:43:15.160008+0000 2024-03-25T19:43:15.159 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:43:15.160008+0000 2024-03-25T19:43:15.159 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:15.159 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:15.159 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:43:15.159 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:43:15.159 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:43:15.159 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:15.160 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.160+0000 7fa8c2332640 1 Processor -- start 2024-03-25T19:43:15.160 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.160+0000 7fa8c2332640 1 -- start start 2024-03-25T19:43:15.160 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.160+0000 7fa8c2332640 10 monclient: init 2024-03-25T19:43:15.160 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.160+0000 7fa8c2332640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:15.160 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.160+0000 7fa8c2332640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:15.161 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.160+0000 7fa8c2332640 10 monclient(hunting): picked mon.noname-c con 0x7fa8bc0a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:43:15.161 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.160+0000 7fa8c2332640 10 monclient(hunting): picked mon.noname-a con 0x7fa8bc13f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:43:15.161 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.160+0000 7fa8c2332640 10 monclient(hunting): picked mon.noname-b con 0x7fa8bc1430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:43:15.161 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.160+0000 7fa8c2332640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8bc142520 con 0x7fa8bc13f710 2024-03-25T19:43:15.161 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.160+0000 7fa8c2332640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8bc0b28a0 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.161 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.160+0000 7fa8c2332640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8bc0b2a30 con 0x7fa8bc1430c0 2024-03-25T19:43:15.161 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.160+0000 7fa8c2332640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:15.161 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.160+0000 7fa8c2332640 10 monclient(hunting): authenticate will time out at 4030.242747s 2024-03-25T19:43:15.170 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.171+0000 7fa8bbfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa8bc0a98f0 0x7fa8bc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:42462/0 (socket says 172.21.15.81:42462) 2024-03-25T19:43:15.170 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.171+0000 7fa8bbfff640 1 -- 172.21.15.81:0/2260859062 learned_addr learned my addr 172.21.15.81:0/2260859062 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:43:15.178 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.179+0000 7fa8baffd640 1 -- 172.21.15.81:0/2260859062 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229106873 0 0) 0x7fa8bc0b28a0 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.179 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.179+0000 7fa8baffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:15.179 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.179+0000 7fa8baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:15.179 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.179+0000 7fa8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:15.179 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.179+0000 7fa8baffd640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:43:15.179 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.179+0000 7fa8baffd640 1 -- 172.21.15.81:0/2260859062 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a4003520 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.183 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.184+0000 7fa8baffd640 1 -- 172.21.15.81:0/2260859062 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127173990 0 0) 0x7fa8bc0b2a30 con 0x7fa8bc1430c0 2024-03-25T19:43:15.183 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.184+0000 7fa8baffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:15.183 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.184+0000 7fa8baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:15.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.184+0000 7fa8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:15.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.184+0000 7fa8baffd640 10 monclient(hunting): my global_id is 4262 2024-03-25T19:43:15.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.184+0000 7fa8baffd640 1 -- 172.21.15.81:0/2260859062 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8bc0b28a0 con 0x7fa8bc1430c0 2024-03-25T19:43:15.189 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.189+0000 7fa8baffd640 1 -- 172.21.15.81:0/2260859062 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929848571 0 0) 0x7fa8bc142520 con 0x7fa8bc13f710 2024-03-25T19:43:15.189 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.189+0000 7fa8baffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:15.189 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.189+0000 7fa8baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:15.189 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.189+0000 7fa8baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:15.189 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.189+0000 7fa8baffd640 10 monclient(hunting): my global_id is 4278 2024-03-25T19:43:15.189 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.189+0000 7fa8baffd640 1 -- 172.21.15.81:0/2260859062 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8bc0b2a30 con 0x7fa8bc13f710 2024-03-25T19:43:15.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.192+0000 7fa8baffd640 1 -- 172.21.15.81:0/2260859062 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1284485328 0 0) 0x7fa8a4003520 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.192+0000 7fa8baffd640 1 -- 172.21.15.81:0/2260859062 >> v1:172.21.15.81:6790/0 conn(0x7fa8bc1430c0 legacy=0x7fa8bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.192+0000 7fa8baffd640 1 -- 172.21.15.81:0/2260859062 >> v1:172.21.15.81:6789/0 conn(0x7fa8bc13f710 legacy=0x7fa8bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.192+0000 7fa8baffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:15.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.192+0000 7fa8baffd640 1 monclient: found mon.noname-c 2024-03-25T19:43:15.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.192+0000 7fa8baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:15.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.192+0000 7fa8baffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:43:15.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.192+0000 7fa8baffd640 1 -- 172.21.15.81:0/2260859062 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8bc0b44c0 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.192+0000 7fa8baffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:15.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.192+0000 7fa8baffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:15.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.192+0000 7fa8baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:15.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.192+0000 7fa8baffd640 1 -- 172.21.15.81:0/2260859062 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa8ac001690 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.192+0000 7fa8baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.192+0000 7fa8baffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:15.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.192+0000 7fa8baffd640 10 monclient: dump: 2024-03-25T19:43:15.193 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:15.193 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:15.193 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:15.193 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:15.193 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.193 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:15.194 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:15.194 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:15.194 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:15.194 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:15.194 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.192+0000 7fa8c2332640 5 monclient: authenticate success, global_id 4279 2024-03-25T19:43:15.194 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.192+0000 7fa8c2332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:15.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.201+0000 7fa8baffd640 1 -- 172.21.15.81:0/2260859062 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8ac0017f0 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.201+0000 7fa8baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:15.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.201+0000 7fa8baffd640 1 -- 172.21.15.81:0/2260859062 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa8ac001c00 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.201+0000 7fa8baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.201+0000 7fa8baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:15.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.201+0000 7fa8baffd640 10 monclient: dump: 2024-03-25T19:43:15.201 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:15.201 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:15.201 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:15.201 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:15.201 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.201 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:15.201 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:15.201 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:15.201 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:15.201 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:15.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.201+0000 7fa8c2332640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:15.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.201+0000 7fa8c2332640 4 set_mon_vals no callback set 2024-03-25T19:43:15.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.201+0000 7fa8c2332640 10 monclient: shutdown 2024-03-25T19:43:15.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.201+0000 7fa8c2332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:15.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.201+0000 7fa8c2332640 1 -- 172.21.15.81:0/2260859062 >> v1:172.21.15.98:6789/0 conn(0x7fa8bc0a98f0 legacy=0x7fa8bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.204 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.201+0000 7fa8c2332640 1 -- 172.21.15.81:0/2260859062 shutdown_connections 2024-03-25T19:43:15.204 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.201+0000 7fa8c2332640 1 -- 172.21.15.81:0/2260859062 >> 172.21.15.81:0/2260859062 conn(0x7fa8bc0b7ea0 msgr2=0x7fa8bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:15.204 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.202+0000 7fa8c2332640 1 -- 172.21.15.81:0/2260859062 shutdown_connections 2024-03-25T19:43:15.204 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.202+0000 7fa8c2332640 1 -- 172.21.15.81:0/2260859062 wait complete. 2024-03-25T19:43:15.204 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.202+0000 7fa8c2332640 10 monclient: build_initial_monmap 2024-03-25T19:43:15.204 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.202+0000 7fa8c2332640 10 monclient: monmap: 2024-03-25T19:43:15.204 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:43:15.204 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:15.204 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:43:15.204 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:43:15.204 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:15.204 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:43:15.204 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:43:15.204 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:43:15.204 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:43:15.205 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:15.205 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.203+0000 7fa8c2332640 1 librados: starting msgr at 2024-03-25T19:43:15.205 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.203+0000 7fa8c2332640 1 librados: starting objecter 2024-03-25T19:43:15.205 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.203+0000 7fa8c2332640 1 Processor -- start 2024-03-25T19:43:15.205 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.203+0000 7fa8c2332640 1 -- start start 2024-03-25T19:43:15.205 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.203+0000 7fa8c2332640 1 librados: setting wanted keys 2024-03-25T19:43:15.205 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.203+0000 7fa8c2332640 1 librados: calling monclient init 2024-03-25T19:43:15.205 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.203+0000 7fa8c2332640 10 monclient: init 2024-03-25T19:43:15.205 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.203+0000 7fa8c2332640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:15.205 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.203+0000 7fa8c2332640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:15.205 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.203+0000 7fa8c2332640 10 monclient(hunting): picked mon.noname-b con 0x7fa8bc0a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:43:15.205 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.203+0000 7fa8c2332640 10 monclient(hunting): picked mon.noname-c con 0x7fa8bc13f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:43:15.205 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.203+0000 7fa8c2332640 10 monclient(hunting): picked mon.noname-a con 0x7fa8bc1430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:43:15.205 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.203+0000 7fa8c2332640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8bc137ab0 con 0x7fa8bc1430c0 2024-03-25T19:43:15.206 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.203+0000 7fa8c2332640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8bc138bd0 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.206 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.203+0000 7fa8c2332640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8bc138e40 con 0x7fa8bc13f710 2024-03-25T19:43:15.206 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.204+0000 7fa8c2332640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:15.206 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.204+0000 7fa8c2332640 10 monclient(hunting): authenticate will time out at 4030.285854s 2024-03-25T19:43:15.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.212+0000 7fa8bbfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa8bc0a98f0 0x7fa8bc137f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:42502/0 (socket says 172.21.15.81:42502) 2024-03-25T19:43:15.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.212+0000 7fa8bbfff640 1 -- 172.21.15.81:0/1387254625 learned_addr learned my addr 172.21.15.81:0/1387254625 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:43:15.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.213+0000 7fa8b8ff9640 10 client.?.objecter ms_handle_connect 0x7fa8bc0a98f0 2024-03-25T19:43:15.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.213+0000 7fa8b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:15.218 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.218+0000 7fa8b8ff9640 1 -- 172.21.15.81:0/1387254625 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590106088 0 0) 0x7fa8bc138bd0 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.218 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.218+0000 7fa8b8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:15.218 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.218+0000 7fa8b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:15.218 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.218+0000 7fa8b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:15.218 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.218+0000 7fa8b8ff9640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:43:15.218 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.218+0000 7fa8b8ff9640 1 -- 172.21.15.81:0/1387254625 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a8004eb0 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.223+0000 7fa8b8ff9640 10 client.?.objecter ms_handle_connect 0x7fa8bc1430c0 2024-03-25T19:43:15.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.224+0000 7fa8b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:15.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.227+0000 7fa8b8ff9640 1 -- 172.21.15.81:0/1387254625 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1904502850 0 0) 0x7fa8a8004eb0 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.227+0000 7fa8b8ff9640 1 -- 172.21.15.81:0/1387254625 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8bc138bd0 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.227+0000 7fa8b8ff9640 1 -- 172.21.15.81:0/1387254625 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa8ac001bc0 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.227+0000 7fa8b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.227+0000 7fa8b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:15.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.227+0000 7fa8b8ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:15.227 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:15.227 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:15.227 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:15.227 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:15.227 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.227 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:15.227 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:15.227 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:15.227 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:15.227 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:15.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.227+0000 7fa8b8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:15.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.232+0000 7fa8b8ff9640 10 client.?.objecter ms_handle_connect 0x7fa8bc13f710 2024-03-25T19:43:15.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.232+0000 7fa8b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:15.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8b8ff9640 1 -- 172.21.15.81:0/1387254625 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2249867915 0 0) 0x7fa8bc138bd0 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8b8ff9640 1 -- 172.21.15.81:0/1387254625 >> v1:172.21.15.81:6790/0 conn(0x7fa8bc13f710 legacy=0x7fa8bc1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8b8ff9640 1 -- 172.21.15.81:0/1387254625 >> v1:172.21.15.81:6789/0 conn(0x7fa8bc1430c0 legacy=0x7fa8bc1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8b8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:43:15.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8b8ff9640 1 monclient: found mon.b 2024-03-25T19:43:15.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:15.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8b8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:43:15.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8b8ff9640 1 -- 172.21.15.81:0/1387254625 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8bc139120 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.241 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8b8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:43:15.241 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8b8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:43:15.241 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:15.241 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8c2332640 5 monclient: authenticate success, global_id 4282 2024-03-25T19:43:15.241 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8c2332640 10 monclient: _renew_subs 2024-03-25T19:43:15.241 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8c2332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:43:15.241 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8c2332640 1 -- 172.21.15.81:0/1387254625 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8bc139400 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.241 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8c2332640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:15.241 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8c2332640 10 monclient: _renew_subs 2024-03-25T19:43:15.241 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8c2332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:43:15.241 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8c2332640 1 -- 172.21.15.81:0/1387254625 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8bc0b7880 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.241 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.240+0000 7fa8c2332640 1 librados: init done 2024-03-25T19:43:15.242 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.241+0000 7fa8c2332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:15.242 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.241+0000 7fa8c2332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:15.242 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.241+0000 7fa8c2332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:43:15.242 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.241+0000 7fa8c2332640 1 -- 172.21.15.81:0/1387254625 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8bc137810 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.517+0000 7fa8b8ff9640 1 -- 172.21.15.81:0/1387254625 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8ac004a90 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.517+0000 7fa8b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:15.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.517+0000 7fa8b8ff9640 1 -- 172.21.15.81:0/1387254625 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa8ac004ea0 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.517+0000 7fa8b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:15.518 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.517+0000 7fa8b8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:15.518 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.517+0000 7fa8b8ff9640 10 monclient: dump: 2024-03-25T19:43:15.518 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:15.518 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:15.518 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:15.518 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:15.518 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:15.518 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:15.518 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:15.518 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:15.518 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:15.518 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:15.518 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.517+0000 7fa8ba7fc640 4 set_mon_vals no callback set 2024-03-25T19:43:15.519 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.518+0000 7fa8b8ff9640 1 -- 172.21.15.81:0/1387254625 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fa8ac0050c0 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.519 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.519+0000 7fa8bb7fe640 10 monclient: get_auth_request con 0x7fa8a806dc70 auth_method 0 2024-03-25T19:43:15.519 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.519+0000 7fa8b8ff9640 10 client.4282.objecter ms_handle_connect 0x7fa8a806dc70 2024-03-25T19:43:15.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.523+0000 7fa8b8ff9640 1 -- 172.21.15.81:0/1387254625 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fa8ac0549b0 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.523+0000 7fa8b8ff9640 10 client.4282.objecter ms_dispatch 0x7fa8bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:15.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.523+0000 7fa8b8ff9640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:15.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.523+0000 7fa8b8ff9640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:15.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.523+0000 7fa8b8ff9640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-25T19:43:15.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.524+0000 7fa8b8ff9640 1 -- 172.21.15.81:0/1387254625 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa8ac0504c0 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.524+0000 7fa8b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:15.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.524+0000 7fa8b8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:15.776 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.777+0000 7fa8c2332640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:15.776 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.777+0000 7fa8c2332640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:15.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.777+0000 7fa8c2332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:43:15.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.777+0000 7fa8c2332640 1 -- 172.21.15.81:0/1387254625 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8bc0627b0 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.853 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.853+0000 7fa8b8ff9640 1 -- 172.21.15.81:0/1387254625 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fa8ac0523f0 con 0x7fa8bc0a98f0 2024-03-25T19:43:15.853 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.853+0000 7fa8b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:15.853 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.853+0000 7fa8b8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:15.853 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:43:15.853 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:15.855 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.855+0000 7fa8c2332640 10 librados: watch_flush enter 2024-03-25T19:43:15.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.855+0000 7fa8c2332640 10 librados: watch_flush exit 2024-03-25T19:43:15.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.855+0000 7fa8c2332640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-25T19:43:15.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.855+0000 7fa8c2332640 10 client.4282.objecter successfully canceled tick 2024-03-25T19:43:15.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.855+0000 7fa8c2332640 1 -- 172.21.15.81:0/1387254625 >> v1:172.21.15.98:6800/3980829775 conn(0x7fa8a806dc70 legacy=0x7fa8a8070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.855+0000 7fa8c2332640 10 monclient: shutdown 2024-03-25T19:43:15.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.855+0000 7fa8c2332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:15.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.855+0000 7fa8c2332640 1 -- 172.21.15.81:0/1387254625 >> v1:172.21.15.98:6789/0 conn(0x7fa8bc0a98f0 legacy=0x7fa8bc137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:15.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.855+0000 7fa8c2332640 1 -- 172.21.15.81:0/1387254625 shutdown_connections 2024-03-25T19:43:15.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.856+0000 7fa8c2332640 1 -- 172.21.15.81:0/1387254625 >> 172.21.15.81:0/1387254625 conn(0x7fa8bc0b7ea0 msgr2=0x7fa8bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:15.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.856+0000 7fa8c2332640 1 -- 172.21.15.81:0/1387254625 shutdown_connections 2024-03-25T19:43:15.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.856+0000 7fa8c2332640 1 -- 172.21.15.81:0/1387254625 wait complete. 2024-03-25T19:43:15.857 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:15.856+0000 7fa8c2332640 1 librados: shutdown 2024-03-25T19:43:15.865 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:16.738 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:01:39.379 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:16.786 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:01:39.428 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:16.822 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:01:39.464 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:16.838 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:01:39.480 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:16.865 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:39.499 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:16.871 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:39.505 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:16.874 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:01:39.516 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:16.888 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:01:39.530 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:16.906 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:39.541 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:22.672 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:22.790 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.790+0000 7fcb7eaf4640 10 monclient: get_monmap_and_config 2024-03-25T19:43:22.790 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.790+0000 7fcb7eaf4640 10 monclient: build_initial_monmap 2024-03-25T19:43:22.790 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.790+0000 7fcb7eaf4640 10 monclient: monmap: 2024-03-25T19:43:22.790 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:43:22.790 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:22.790 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:43:22.791720+0000 2024-03-25T19:43:22.790 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:43:22.791720+0000 2024-03-25T19:43:22.790 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:22.791 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:22.791 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:43:22.791 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:43:22.791 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:43:22.791 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:22.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.791+0000 7fcb7eaf4640 1 Processor -- start 2024-03-25T19:43:22.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.791+0000 7fcb7eaf4640 1 -- start start 2024-03-25T19:43:22.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.791+0000 7fcb7eaf4640 10 monclient: init 2024-03-25T19:43:22.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.792+0000 7fcb7eaf4640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:22.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.792+0000 7fcb7eaf4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:22.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.792+0000 7fcb7eaf4640 10 monclient(hunting): picked mon.noname-c con 0x7fcb780a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:43:22.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.792+0000 7fcb7eaf4640 10 monclient(hunting): picked mon.noname-a con 0x7fcb7813f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:43:22.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.792+0000 7fcb7eaf4640 10 monclient(hunting): picked mon.noname-b con 0x7fcb781430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:43:22.793 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.792+0000 7fcb7eaf4640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb78142520 con 0x7fcb7813f710 2024-03-25T19:43:22.793 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.792+0000 7fcb7eaf4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb780b28a0 con 0x7fcb780a98f0 2024-03-25T19:43:22.793 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.792+0000 7fcb7eaf4640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb780b2a30 con 0x7fcb781430c0 2024-03-25T19:43:22.793 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.792+0000 7fcb7eaf4640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:22.793 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.792+0000 7fcb7eaf4640 10 monclient(hunting): authenticate will time out at 4037.874479s 2024-03-25T19:43:22.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.801+0000 7fcb7d091640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fcb781430c0 0x7fcb7814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:54346/0 (socket says 172.21.15.81:54346) 2024-03-25T19:43:22.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.801+0000 7fcb7d091640 1 -- 172.21.15.81:0/316596881 learned_addr learned my addr 172.21.15.81:0/316596881 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:43:22.814 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.815+0000 7fcb777fe640 1 -- 172.21.15.81:0/316596881 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154222237 0 0) 0x7fcb780b2a30 con 0x7fcb781430c0 2024-03-25T19:43:22.814 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.815+0000 7fcb777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:22.815 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.815+0000 7fcb777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:22.815 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.815+0000 7fcb777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:22.815 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.815+0000 7fcb777fe640 10 monclient(hunting): my global_id is 4268 2024-03-25T19:43:22.815 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.815+0000 7fcb777fe640 1 -- 172.21.15.81:0/316596881 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb68003520 con 0x7fcb781430c0 2024-03-25T19:43:22.817 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.817+0000 7fcb777fe640 1 -- 172.21.15.81:0/316596881 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029585209 0 0) 0x7fcb780b28a0 con 0x7fcb780a98f0 2024-03-25T19:43:22.817 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.818+0000 7fcb777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:22.817 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.818+0000 7fcb777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:22.817 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.818+0000 7fcb777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:22.817 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.818+0000 7fcb777fe640 10 monclient(hunting): my global_id is 4285 2024-03-25T19:43:22.817 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.818+0000 7fcb777fe640 1 -- 172.21.15.81:0/316596881 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb780b2a30 con 0x7fcb780a98f0 2024-03-25T19:43:22.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb777fe640 1 -- 172.21.15.81:0/316596881 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4124922419 0 0) 0x7fcb68003520 con 0x7fcb781430c0 2024-03-25T19:43:22.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb777fe640 1 -- 172.21.15.81:0/316596881 >> v1:172.21.15.98:6789/0 conn(0x7fcb780a98f0 legacy=0x7fcb780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:22.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb777fe640 1 -- 172.21.15.81:0/316596881 >> v1:172.21.15.81:6789/0 conn(0x7fcb7813f710 legacy=0x7fcb781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:22.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb777fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:22.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb777fe640 1 monclient: found mon.noname-b 2024-03-25T19:43:22.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:22.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb777fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:43:22.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb777fe640 1 -- 172.21.15.81:0/316596881 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb780b44c0 con 0x7fcb781430c0 2024-03-25T19:43:22.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb777fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:22.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb777fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:22.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:22.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb777fe640 1 -- 172.21.15.81:0/316596881 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fcb70001a10 con 0x7fcb781430c0 2024-03-25T19:43:22.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb7eaf4640 5 monclient: authenticate success, global_id 4268 2024-03-25T19:43:22.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb7eaf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:22.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:22.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:22.825 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb777fe640 10 monclient: dump: 2024-03-25T19:43:22.825 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:22.825 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:22.825 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:22.825 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:22.825 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:22.825 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:22.825 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:22.825 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:22.825 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:22.825 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:22.825 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.823+0000 7fcb7eaf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:22.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.848+0000 7fcb777fe640 1 -- 172.21.15.81:0/316596881 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb70001c30 con 0x7fcb781430c0 2024-03-25T19:43:22.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.848+0000 7fcb777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:22.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.848+0000 7fcb777fe640 1 -- 172.21.15.81:0/316596881 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fcb70001fc0 con 0x7fcb781430c0 2024-03-25T19:43:22.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.848+0000 7fcb777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:22.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.848+0000 7fcb777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:22.848 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.848+0000 7fcb777fe640 10 monclient: dump: 2024-03-25T19:43:22.848 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:22.848 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:22.848 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:22.848 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:22.848 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:22.848 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:22.848 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:22.848 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:22.848 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:22.848 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:22.848 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.848+0000 7fcb7eaf4640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:22.848 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.848+0000 7fcb7eaf4640 4 set_mon_vals no callback set 2024-03-25T19:43:22.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.849+0000 7fcb7eaf4640 10 monclient: shutdown 2024-03-25T19:43:22.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.849+0000 7fcb7eaf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:22.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.849+0000 7fcb7eaf4640 1 -- 172.21.15.81:0/316596881 >> v1:172.21.15.81:6790/0 conn(0x7fcb781430c0 legacy=0x7fcb7814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:22.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.849+0000 7fcb7eaf4640 1 -- 172.21.15.81:0/316596881 shutdown_connections 2024-03-25T19:43:22.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.849+0000 7fcb7eaf4640 1 -- 172.21.15.81:0/316596881 >> 172.21.15.81:0/316596881 conn(0x7fcb780b7ea0 msgr2=0x7fcb780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:22.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.849+0000 7fcb7eaf4640 1 -- 172.21.15.81:0/316596881 shutdown_connections 2024-03-25T19:43:22.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.849+0000 7fcb7eaf4640 1 -- 172.21.15.81:0/316596881 wait complete. 2024-03-25T19:43:22.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.849+0000 7fcb7eaf4640 10 monclient: build_initial_monmap 2024-03-25T19:43:22.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.850+0000 7fcb7eaf4640 10 monclient: monmap: 2024-03-25T19:43:22.850 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:43:22.850 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:22.850 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:43:22.851 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:43:22.851 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:22.851 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:43:22.851 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:43:22.851 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:43:22.851 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:43:22.851 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:22.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.850+0000 7fcb7eaf4640 1 librados: starting msgr at 2024-03-25T19:43:22.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.850+0000 7fcb7eaf4640 1 librados: starting objecter 2024-03-25T19:43:22.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.850+0000 7fcb7eaf4640 1 Processor -- start 2024-03-25T19:43:22.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.850+0000 7fcb7eaf4640 1 -- start start 2024-03-25T19:43:22.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.850+0000 7fcb7eaf4640 1 librados: setting wanted keys 2024-03-25T19:43:22.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.850+0000 7fcb7eaf4640 1 librados: calling monclient init 2024-03-25T19:43:22.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.850+0000 7fcb7eaf4640 10 monclient: init 2024-03-25T19:43:22.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.851+0000 7fcb7eaf4640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:22.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.851+0000 7fcb7eaf4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:22.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.851+0000 7fcb7eaf4640 10 monclient(hunting): picked mon.noname-a con 0x7fcb780a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:43:22.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.851+0000 7fcb7eaf4640 10 monclient(hunting): picked mon.noname-b con 0x7fcb7813f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:43:22.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.851+0000 7fcb7eaf4640 10 monclient(hunting): picked mon.noname-c con 0x7fcb781430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:43:22.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.851+0000 7fcb7eaf4640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb781dc070 con 0x7fcb780a98f0 2024-03-25T19:43:22.853 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.851+0000 7fcb7eaf4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb78136bd0 con 0x7fcb7813f710 2024-03-25T19:43:22.853 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.851+0000 7fcb7eaf4640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb78136eb0 con 0x7fcb781430c0 2024-03-25T19:43:22.853 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.851+0000 7fcb7eaf4640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:22.853 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.851+0000 7fcb7eaf4640 10 monclient(hunting): authenticate will time out at 4037.933548s 2024-03-25T19:43:22.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.861+0000 7fcb7d091640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fcb781430c0 0x7fcb781db960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:54396/0 (socket says 172.21.15.81:54396) 2024-03-25T19:43:22.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.861+0000 7fcb7d091640 1 -- 172.21.15.81:0/1016832858 learned_addr learned my addr 172.21.15.81:0/1016832858 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:43:22.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.862+0000 7fcb757fa640 10 client.?.objecter ms_handle_connect 0x7fcb781430c0 2024-03-25T19:43:22.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.862+0000 7fcb757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:22.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.865+0000 7fcb757fa640 10 client.?.objecter ms_handle_connect 0x7fcb780a98f0 2024-03-25T19:43:22.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.865+0000 7fcb757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:22.869 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.869+0000 7fcb757fa640 10 client.?.objecter ms_handle_connect 0x7fcb7813f710 2024-03-25T19:43:22.869 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.869+0000 7fcb757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:22.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.872+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2105020920 0 0) 0x7fcb78136eb0 con 0x7fcb781430c0 2024-03-25T19:43:22.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.872+0000 7fcb757fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:22.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.872+0000 7fcb757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:22.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.872+0000 7fcb757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:22.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.872+0000 7fcb757fa640 10 monclient(hunting): my global_id is 4271 2024-03-25T19:43:22.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.872+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb5c004eb0 con 0x7fcb781430c0 2024-03-25T19:43:22.873 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.874+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000984534 0 0) 0x7fcb78136bd0 con 0x7fcb7813f710 2024-03-25T19:43:22.873 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.874+0000 7fcb757fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:22.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.874+0000 7fcb757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:22.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.874+0000 7fcb757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:22.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.874+0000 7fcb757fa640 10 monclient(hunting): my global_id is 4288 2024-03-25T19:43:22.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.874+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb78136eb0 con 0x7fcb7813f710 2024-03-25T19:43:22.882 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.882+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 73703466 0 0) 0x7fcb78136eb0 con 0x7fcb7813f710 2024-03-25T19:43:22.882 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.882+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb78136bd0 con 0x7fcb7813f710 2024-03-25T19:43:22.882 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.882+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fcb640018a0 con 0x7fcb7813f710 2024-03-25T19:43:22.882 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.882+0000 7fcb757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:22.882 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.882+0000 7fcb757fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:22.882 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.882+0000 7fcb757fa640 10 monclient(hunting): dump: 2024-03-25T19:43:22.882 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:22.882 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:22.882 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:22.882 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:22.883 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:22.883 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:22.883 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:22.883 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:22.883 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:22.883 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:22.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.882+0000 7fcb757fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:22.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.882+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903112814 0 0) 0x7fcb781dc070 con 0x7fcb780a98f0 2024-03-25T19:43:22.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.882+0000 7fcb757fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:22.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.882+0000 7fcb757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:22.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.882+0000 7fcb757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:22.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.882+0000 7fcb757fa640 10 monclient(hunting): my global_id is 4287 2024-03-25T19:43:22.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.883+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb78136eb0 con 0x7fcb780a98f0 2024-03-25T19:43:22.892 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.892+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2768972837 0 0) 0x7fcb78136bd0 con 0x7fcb7813f710 2024-03-25T19:43:22.892 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.892+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 >> v1:172.21.15.81:6790/0 conn(0x7fcb781430c0 legacy=0x7fcb781db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:22.892 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.892+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 >> v1:172.21.15.81:6789/0 conn(0x7fcb780a98f0 legacy=0x7fcb78137890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:22.892 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.892+0000 7fcb757fa640 10 monclient: _finish_hunting 0 2024-03-25T19:43:22.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.892+0000 7fcb757fa640 1 monclient: found mon.b 2024-03-25T19:43:22.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.892+0000 7fcb757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:22.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.892+0000 7fcb757fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:43:22.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.892+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb78137190 con 0x7fcb7813f710 2024-03-25T19:43:22.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.892+0000 7fcb757fa640 10 monclient: _finish_auth 0 2024-03-25T19:43:22.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.892+0000 7fcb757fa640 10 monclient: _check_auth_tickets 2024-03-25T19:43:22.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.892+0000 7fcb757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:22.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.892+0000 7fcb7eaf4640 5 monclient: authenticate success, global_id 4288 2024-03-25T19:43:22.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.892+0000 7fcb7eaf4640 10 monclient: _renew_subs 2024-03-25T19:43:22.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.893+0000 7fcb7eaf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:43:22.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.893+0000 7fcb7eaf4640 1 -- 172.21.15.81:0/1016832858 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb78137470 con 0x7fcb7813f710 2024-03-25T19:43:22.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.893+0000 7fcb7eaf4640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:22.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.893+0000 7fcb7eaf4640 10 monclient: _renew_subs 2024-03-25T19:43:22.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.893+0000 7fcb7eaf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:43:22.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.893+0000 7fcb7eaf4640 1 -- 172.21.15.81:0/1016832858 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb781355e0 con 0x7fcb7813f710 2024-03-25T19:43:22.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.893+0000 7fcb7eaf4640 1 librados: init done 2024-03-25T19:43:22.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.893+0000 7fcb7eaf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:22.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.893+0000 7fcb7eaf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:22.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.893+0000 7fcb7eaf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:43:22.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:22.893+0000 7fcb7eaf4640 1 -- 172.21.15.81:0/1016832858 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb78135830 con 0x7fcb7813f710 2024-03-25T19:43:23.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.175+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb64001d50 con 0x7fcb7813f710 2024-03-25T19:43:23.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.175+0000 7fcb757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:23.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.175+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fcb640037d0 con 0x7fcb7813f710 2024-03-25T19:43:23.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.175+0000 7fcb757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:23.175 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.175+0000 7fcb757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:23.175 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.175+0000 7fcb757fa640 10 monclient: dump: 2024-03-25T19:43:23.175 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:23.175 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:23.175 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:23.175 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:23.175 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:23.175 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:23.175 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:23.175 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:23.175 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:23.175 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:23.175 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.175+0000 7fcb76ffd640 4 set_mon_vals no callback set 2024-03-25T19:43:23.176 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.175+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fcb64003970 con 0x7fcb7813f710 2024-03-25T19:43:23.176 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.176+0000 7fcb7c890640 10 monclient: get_auth_request con 0x7fcb5c06dbb0 auth_method 0 2024-03-25T19:43:23.176 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.176+0000 7fcb757fa640 10 client.4288.objecter ms_handle_connect 0x7fcb5c06dbb0 2024-03-25T19:43:23.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.180+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fcb64053540 con 0x7fcb7813f710 2024-03-25T19:43:23.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.180+0000 7fcb757fa640 10 client.4288.objecter ms_dispatch 0x7fcb78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:23.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.180+0000 7fcb757fa640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:23.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.180+0000 7fcb757fa640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:23.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.181+0000 7fcb757fa640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T19:43:23.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.181+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb6404ee80 con 0x7fcb7813f710 2024-03-25T19:43:23.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.181+0000 7fcb757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:23.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.181+0000 7fcb757fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:23.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.426+0000 7fcb7eaf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:23.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.426+0000 7fcb7eaf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:23.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.426+0000 7fcb7eaf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:43:23.426 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.426+0000 7fcb7eaf4640 1 -- 172.21.15.81:0/1016832858 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb780627b0 con 0x7fcb7813f710 2024-03-25T19:43:23.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.464+0000 7fcb757fa640 1 -- 172.21.15.81:0/1016832858 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fcb64056ba0 con 0x7fcb7813f710 2024-03-25T19:43:23.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.464+0000 7fcb757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:23.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.464+0000 7fcb757fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:23.464 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:43:23.464 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:23.467 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.467+0000 7fcb7eaf4640 10 librados: watch_flush enter 2024-03-25T19:43:23.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.467+0000 7fcb7eaf4640 10 librados: watch_flush exit 2024-03-25T19:43:23.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.467+0000 7fcb7eaf4640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T19:43:23.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.467+0000 7fcb7eaf4640 10 client.4288.objecter successfully canceled tick 2024-03-25T19:43:23.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.467+0000 7fcb7eaf4640 1 -- 172.21.15.81:0/1016832858 >> v1:172.21.15.98:6800/3980829775 conn(0x7fcb5c06dbb0 legacy=0x7fcb5c070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:23.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.467+0000 7fcb7eaf4640 10 monclient: shutdown 2024-03-25T19:43:23.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.467+0000 7fcb7eaf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:23.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.467+0000 7fcb7eaf4640 1 -- 172.21.15.81:0/1016832858 >> v1:172.21.15.98:6789/0 conn(0x7fcb7813f710 legacy=0x7fcb781db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:23.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.468+0000 7fcb7eaf4640 1 -- 172.21.15.81:0/1016832858 shutdown_connections 2024-03-25T19:43:23.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.468+0000 7fcb7eaf4640 1 -- 172.21.15.81:0/1016832858 >> 172.21.15.81:0/1016832858 conn(0x7fcb780b7ea0 msgr2=0x7fcb7813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:23.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.468+0000 7fcb7eaf4640 1 -- 172.21.15.81:0/1016832858 shutdown_connections 2024-03-25T19:43:23.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.468+0000 7fcb7eaf4640 1 -- 172.21.15.81:0/1016832858 wait complete. 2024-03-25T19:43:23.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:23.468+0000 7fcb7eaf4640 1 librados: shutdown 2024-03-25T19:43:23.479 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:30.286 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:30.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.402+0000 7f2b15493640 10 monclient: get_monmap_and_config 2024-03-25T19:43:30.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.402+0000 7f2b15493640 10 monclient: build_initial_monmap 2024-03-25T19:43:30.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.402+0000 7f2b15493640 10 monclient: monmap: 2024-03-25T19:43:30.403 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:43:30.403 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:30.403 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:43:30.404671+0000 2024-03-25T19:43:30.403 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:43:30.404671+0000 2024-03-25T19:43:30.403 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:30.403 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:30.403 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:43:30.404 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:43:30.404 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:43:30.404 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:30.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.403+0000 7f2b15493640 1 Processor -- start 2024-03-25T19:43:30.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.403+0000 7f2b15493640 1 -- start start 2024-03-25T19:43:30.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.403+0000 7f2b15493640 10 monclient: init 2024-03-25T19:43:30.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.404+0000 7f2b15493640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:30.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.404+0000 7f2b15493640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:43:30.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.404+0000 7f2b15493640 10 monclient(hunting): picked mon.noname-b con 0x7f2b1006c380 addr v1:172.21.15.81:6790/0 2024-03-25T19:43:30.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.404+0000 7f2b15493640 10 monclient(hunting): picked mon.noname-a con 0x7f2b100b12d0 addr v1:172.21.15.81:6789/0 2024-03-25T19:43:30.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.404+0000 7f2b15493640 10 monclient(hunting): picked mon.noname-c con 0x7f2b100aedf0 addr v1:172.21.15.98:6789/0 2024-03-25T19:43:30.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.404+0000 7f2b15493640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b100b16f0 con 0x7f2b100b12d0 2024-03-25T19:43:30.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.404+0000 7f2b15493640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b100adfe0 con 0x7f2b100aedf0 2024-03-25T19:43:30.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.404+0000 7f2b15493640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b100add00 con 0x7f2b1006c380 2024-03-25T19:43:30.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.404+0000 7f2b15493640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:30.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.404+0000 7f2b15493640 10 monclient(hunting): authenticate will time out at 4045.487266s 2024-03-25T19:43:30.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.409+0000 7f2b0effd640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f2b1006c380 0x7f2b100b0bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:57196/0 (socket says 172.21.15.81:57196) 2024-03-25T19:43:30.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.409+0000 7f2b0effd640 1 -- 172.21.15.81:0/4286966866 learned_addr learned my addr 172.21.15.81:0/4286966866 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:43:30.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.428+0000 7f2b0dffb640 1 -- 172.21.15.81:0/4286966866 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510275032 0 0) 0x7f2b100adfe0 con 0x7f2b100aedf0 2024-03-25T19:43:30.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.428+0000 7f2b0dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.428+0000 7f2b0dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.428+0000 7f2b0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.428+0000 7f2b0dffb640 10 monclient(hunting): my global_id is 4291 2024-03-25T19:43:30.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.428+0000 7f2b0dffb640 1 -- 172.21.15.81:0/4286966866 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af8003520 con 0x7f2b100aedf0 2024-03-25T19:43:30.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.433+0000 7f2b0dffb640 1 -- 172.21.15.81:0/4286966866 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293557276 0 0) 0x7f2b100add00 con 0x7f2b1006c380 2024-03-25T19:43:30.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.433+0000 7f2b0dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.433+0000 7f2b0dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.433+0000 7f2b0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.433+0000 7f2b0dffb640 10 monclient(hunting): my global_id is 4274 2024-03-25T19:43:30.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.433+0000 7f2b0dffb640 1 -- 172.21.15.81:0/4286966866 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b100adfe0 con 0x7f2b1006c380 2024-03-25T19:43:30.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.454+0000 7f2b0dffb640 1 -- 172.21.15.81:0/4286966866 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189813692 0 0) 0x7f2b100b16f0 con 0x7f2b100b12d0 2024-03-25T19:43:30.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.454+0000 7f2b0dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.455 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.454+0000 7f2b0dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.455 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.454+0000 7f2b0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.455 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.454+0000 7f2b0dffb640 10 monclient(hunting): my global_id is 4290 2024-03-25T19:43:30.455 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.454+0000 7f2b0dffb640 1 -- 172.21.15.81:0/4286966866 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b100add00 con 0x7f2b100b12d0 2024-03-25T19:43:30.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b0dffb640 1 -- 172.21.15.81:0/4286966866 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 616264568 0 0) 0x7f2b100add00 con 0x7f2b100b12d0 2024-03-25T19:43:30.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b0dffb640 1 -- 172.21.15.81:0/4286966866 >> v1:172.21.15.81:6790/0 conn(0x7f2b1006c380 legacy=0x7f2b100b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b0dffb640 1 -- 172.21.15.81:0/4286966866 >> v1:172.21.15.98:6789/0 conn(0x7f2b100aedf0 legacy=0x7f2b1014e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b0dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:43:30.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b0dffb640 1 monclient: found mon.noname-a 2024-03-25T19:43:30.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b0dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:30.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b0dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:43:30.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b0dffb640 1 -- 172.21.15.81:0/4286966866 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b100ae2c0 con 0x7f2b100b12d0 2024-03-25T19:43:30.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b0dffb640 10 monclient: _finish_auth 0 2024-03-25T19:43:30.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b0dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:43:30.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:30.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b0dffb640 1 -- 172.21.15.81:0/4286966866 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f2b040015b0 con 0x7f2b100b12d0 2024-03-25T19:43:30.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b15493640 5 monclient: authenticate success, global_id 4290 2024-03-25T19:43:30.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b15493640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:30.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b0dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:30.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b0dffb640 10 monclient: dump: 2024-03-25T19:43:30.481 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:30.481 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:30.481 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:30.481 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:30.481 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.481 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:30.481 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:30.481 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:30.481 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:30.481 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:30.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.478+0000 7f2b15493640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:30.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.549+0000 7f2b0dffb640 1 -- 172.21.15.81:0/4286966866 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b04001710 con 0x7f2b100b12d0 2024-03-25T19:43:30.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.549+0000 7f2b0dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:30.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.549+0000 7f2b0dffb640 1 -- 172.21.15.81:0/4286966866 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f2b04001b60 con 0x7f2b100b12d0 2024-03-25T19:43:30.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.549+0000 7f2b15493640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:30.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.549+0000 7f2b15493640 4 set_mon_vals no callback set 2024-03-25T19:43:30.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.549+0000 7f2b0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.549+0000 7f2b0dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:30.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.549+0000 7f2b0dffb640 10 monclient: dump: 2024-03-25T19:43:30.550 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:30.551 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:30.551 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:30.551 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:30.551 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.551 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:30.551 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:30.551 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:30.551 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:30.551 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:30.552 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.550+0000 7f2b15493640 10 monclient: shutdown 2024-03-25T19:43:30.552 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.550+0000 7f2b15493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:30.552 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.550+0000 7f2b15493640 1 -- 172.21.15.81:0/4286966866 >> v1:172.21.15.81:6789/0 conn(0x7f2b100b12d0 legacy=0x7f2b100ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.552 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.550+0000 7f2b15493640 1 -- 172.21.15.81:0/4286966866 shutdown_connections 2024-03-25T19:43:30.552 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.550+0000 7f2b15493640 1 -- 172.21.15.81:0/4286966866 >> 172.21.15.81:0/4286966866 conn(0x7f2b100b5d10 msgr2=0x7f2b100b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:30.552 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.550+0000 7f2b15493640 1 -- 172.21.15.81:0/4286966866 shutdown_connections 2024-03-25T19:43:30.552 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.550+0000 7f2b15493640 1 -- 172.21.15.81:0/4286966866 wait complete. 2024-03-25T19:43:30.553 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.551+0000 7f2b15493640 10 monclient: build_initial_monmap 2024-03-25T19:43:30.553 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.551+0000 7f2b15493640 10 monclient: monmap: 2024-03-25T19:43:30.553 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:43:30.553 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:30.553 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:43:30.553 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:43:30.553 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:30.553 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:43:30.553 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:43:30.553 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:43:30.553 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:43:30.553 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:30.553 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.552+0000 7f2b15493640 1 librados: starting msgr at 2024-03-25T19:43:30.553 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.552+0000 7f2b15493640 1 librados: starting objecter 2024-03-25T19:43:30.554 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.552+0000 7f2b15493640 1 Processor -- start 2024-03-25T19:43:30.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.552+0000 7f2b15493640 1 -- start start 2024-03-25T19:43:30.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.552+0000 7f2b15493640 1 librados: setting wanted keys 2024-03-25T19:43:30.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.552+0000 7f2b15493640 1 librados: calling monclient init 2024-03-25T19:43:30.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.552+0000 7f2b15493640 10 monclient: init 2024-03-25T19:43:30.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.553+0000 7f2b15493640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:30.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.553+0000 7f2b15493640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:30.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.553+0000 7f2b15493640 10 monclient(hunting): picked mon.noname-b con 0x7f2b1006c380 addr v1:172.21.15.98:6789/0 2024-03-25T19:43:30.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.553+0000 7f2b15493640 10 monclient(hunting): picked mon.noname-a con 0x7f2b100aedf0 addr v1:172.21.15.81:6789/0 2024-03-25T19:43:30.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.553+0000 7f2b15493640 10 monclient(hunting): picked mon.noname-c con 0x7f2b100b12d0 addr v1:172.21.15.81:6790/0 2024-03-25T19:43:30.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.553+0000 7f2b15493640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1006c8e0 con 0x7f2b100aedf0 2024-03-25T19:43:30.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.553+0000 7f2b15493640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b101a5250 con 0x7f2b1006c380 2024-03-25T19:43:30.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.553+0000 7f2b15493640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b101a2c90 con 0x7f2b100b12d0 2024-03-25T19:43:30.556 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.553+0000 7f2b15493640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:30.556 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.553+0000 7f2b15493640 10 monclient(hunting): authenticate will time out at 4045.635960s 2024-03-25T19:43:30.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.568+0000 7f2b0effd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f2b1006c380 0x7f2b101c3e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:56504/0 (socket says 172.21.15.81:56504) 2024-03-25T19:43:30.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.568+0000 7f2b0effd640 1 -- 172.21.15.81:0/2005055404 learned_addr learned my addr 172.21.15.81:0/2005055404 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:43:30.571 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.570+0000 7f2af77fe640 10 client.?.objecter ms_handle_connect 0x7f2b1006c380 2024-03-25T19:43:30.571 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.570+0000 7f2af77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:30.575 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.575+0000 7f2af77fe640 10 client.?.objecter ms_handle_connect 0x7f2b100b12d0 2024-03-25T19:43:30.575 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.575+0000 7f2af77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:30.577 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.576+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727239186 0 0) 0x7f2b101a5250 con 0x7f2b1006c380 2024-03-25T19:43:30.577 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.576+0000 7f2af77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.577 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.576+0000 7f2af77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.577 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.576+0000 7f2af77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.577 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.576+0000 7f2af77fe640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:43:30.577 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.576+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2afc004eb0 con 0x7f2b1006c380 2024-03-25T19:43:30.577 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.576+0000 7f2af77fe640 10 client.?.objecter ms_handle_connect 0x7f2b100aedf0 2024-03-25T19:43:30.577 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.576+0000 7f2af77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:30.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.581+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309743144 0 0) 0x7f2b1006c8e0 con 0x7f2b100aedf0 2024-03-25T19:43:30.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.581+0000 7f2af77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.581+0000 7f2af77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.581+0000 7f2af77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.581+0000 7f2af77fe640 10 monclient(hunting): my global_id is 4293 2024-03-25T19:43:30.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.581+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b101a5250 con 0x7f2b100aedf0 2024-03-25T19:43:30.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.589+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 874276150 0 0) 0x7f2afc004eb0 con 0x7f2b1006c380 2024-03-25T19:43:30.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.589+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b1006c8e0 con 0x7f2b1006c380 2024-03-25T19:43:30.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.589+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f2b000019b0 con 0x7f2b1006c380 2024-03-25T19:43:30.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.589+0000 7f2af77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.589+0000 7f2af77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:30.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.589+0000 7f2af77fe640 10 monclient(hunting): dump: 2024-03-25T19:43:30.591 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:30.591 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:30.591 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:30.591 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:30.591 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.591 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:30.591 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:30.591 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:30.591 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:30.591 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:30.591 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.589+0000 7f2af77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:30.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.591+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515606489 0 0) 0x7f2b101a2c90 con 0x7f2b100b12d0 2024-03-25T19:43:30.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.591+0000 7f2af77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:30.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.591+0000 7f2af77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:30.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.591+0000 7f2af77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:30.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.591+0000 7f2af77fe640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:43:30.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.591+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2afc004eb0 con 0x7f2b100b12d0 2024-03-25T19:43:30.594 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.594+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3459832298 0 0) 0x7f2b101a5250 con 0x7f2b100aedf0 2024-03-25T19:43:30.594 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.594+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b101a2c90 con 0x7f2b100aedf0 2024-03-25T19:43:30.595 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.594+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f2b04001ae0 con 0x7f2b100aedf0 2024-03-25T19:43:30.595 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.594+0000 7f2af77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.595 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.594+0000 7f2af77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:30.595 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.594+0000 7f2af77fe640 10 monclient(hunting): dump: 2024-03-25T19:43:30.595 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:30.595 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:30.595 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:30.595 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:30.595 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.595 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:30.595 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:30.595 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:30.595 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:30.595 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:30.603 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3288737560 0 0) 0x7f2b1006c8e0 con 0x7f2b1006c380 2024-03-25T19:43:30.603 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 >> v1:172.21.15.81:6790/0 conn(0x7f2b100b12d0 legacy=0x7f2b101d7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 >> v1:172.21.15.81:6789/0 conn(0x7f2b100aedf0 legacy=0x7f2b101d76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2af77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:30.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2af77fe640 1 monclient: found mon.b 2024-03-25T19:43:30.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2af77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:30.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2af77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:43:30.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b101c4520 con 0x7f2b1006c380 2024-03-25T19:43:30.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2af77fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:30.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2af77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:30.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2af77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:30.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2b15493640 5 monclient: authenticate success, global_id 4294 2024-03-25T19:43:30.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2b15493640 10 monclient: _renew_subs 2024-03-25T19:43:30.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2b15493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:43:30.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2b15493640 1 -- 172.21.15.81:0/2005055404 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b101c47d0 con 0x7f2b1006c380 2024-03-25T19:43:30.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2b15493640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:30.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2b15493640 10 monclient: _renew_subs 2024-03-25T19:43:30.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2b15493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:43:30.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2b15493640 1 -- 172.21.15.81:0/2005055404 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b101a3390 con 0x7f2b1006c380 2024-03-25T19:43:30.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.602+0000 7f2b15493640 1 librados: init done 2024-03-25T19:43:30.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.603+0000 7f2b15493640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:30.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.603+0000 7f2b15493640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:30.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.603+0000 7f2b15493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:43:30.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.603+0000 7f2b15493640 1 -- 172.21.15.81:0/2005055404 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b101a50c0 con 0x7f2b1006c380 2024-03-25T19:43:30.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.873+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b000047e0 con 0x7f2b1006c380 2024-03-25T19:43:30.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.873+0000 7f2af77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:30.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.873+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f2b00004af0 con 0x7f2b1006c380 2024-03-25T19:43:30.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.873+0000 7f2af77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:30.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.873+0000 7f2af77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:30.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.873+0000 7f2af77fe640 10 monclient: dump: 2024-03-25T19:43:30.875 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:30.875 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:30.875 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:30.875 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:30.875 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:30.875 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:30.875 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:30.875 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:30.875 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:30.875 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:30.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.873+0000 7f2b0d7fa640 4 set_mon_vals no callback set 2024-03-25T19:43:30.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.874+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f2b0001ab00 con 0x7f2b1006c380 2024-03-25T19:43:30.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.875+0000 7f2b0e7fc640 10 monclient: get_auth_request con 0x7f2afc06dfb0 auth_method 0 2024-03-25T19:43:30.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.875+0000 7f2af77fe640 10 client.4294.objecter ms_handle_connect 0x7f2afc06dfb0 2024-03-25T19:43:30.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.879+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f2b00054870 con 0x7f2b1006c380 2024-03-25T19:43:30.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.879+0000 7f2af77fe640 10 client.4294.objecter ms_dispatch 0x7f2b10000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:30.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.879+0000 7f2af77fe640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:30.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.879+0000 7f2af77fe640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:30.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.879+0000 7f2af77fe640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-25T19:43:30.881 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.880+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b000501b0 con 0x7f2b1006c380 2024-03-25T19:43:30.881 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.880+0000 7f2af77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:30.881 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:30.880+0000 7f2af77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:31.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.131+0000 7f2b15493640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:31.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.131+0000 7f2b15493640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:31.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.131+0000 7f2b15493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:43:31.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.131+0000 7f2b15493640 1 -- 172.21.15.81:0/2005055404 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b100627b0 con 0x7f2b1006c380 2024-03-25T19:43:31.170 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.170+0000 7f2af77fe640 1 -- 172.21.15.81:0/2005055404 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f2b000522b0 con 0x7f2b1006c380 2024-03-25T19:43:31.170 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.170+0000 7f2af77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:31.170 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.170+0000 7f2af77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:31.171 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:43:31.171 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:31.173 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.172+0000 7f2b15493640 10 librados: watch_flush enter 2024-03-25T19:43:31.173 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.172+0000 7f2b15493640 10 librados: watch_flush exit 2024-03-25T19:43:31.173 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.172+0000 7f2b15493640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-25T19:43:31.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.172+0000 7f2b15493640 10 client.4294.objecter successfully canceled tick 2024-03-25T19:43:31.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.172+0000 7f2b15493640 1 -- 172.21.15.81:0/2005055404 >> v1:172.21.15.98:6800/3980829775 conn(0x7f2afc06dfb0 legacy=0x7f2afc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:31.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.172+0000 7f2b15493640 10 monclient: shutdown 2024-03-25T19:43:31.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.172+0000 7f2b15493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:31.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.172+0000 7f2b15493640 1 -- 172.21.15.81:0/2005055404 >> v1:172.21.15.98:6789/0 conn(0x7f2b1006c380 legacy=0x7f2b101c3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:31.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.172+0000 7f2b15493640 1 -- 172.21.15.81:0/2005055404 shutdown_connections 2024-03-25T19:43:31.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.172+0000 7f2b15493640 1 -- 172.21.15.81:0/2005055404 >> 172.21.15.81:0/2005055404 conn(0x7f2b100b5d10 msgr2=0x7f2b101c9e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:31.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.172+0000 7f2b15493640 1 -- 172.21.15.81:0/2005055404 shutdown_connections 2024-03-25T19:43:31.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.172+0000 7f2b15493640 1 -- 172.21.15.81:0/2005055404 wait complete. 2024-03-25T19:43:31.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:31.172+0000 7f2b15493640 1 librados: shutdown 2024-03-25T19:43:31.185 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:36.854 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:01:59.495 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:36.862 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:01:59.504 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:36.898 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:01:59.540 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:36.906 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:59.540 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:36.911 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:59.545 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:36.934 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:01:59.569 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:36.935 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:01:59.576 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:36.941 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:01:59.583 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:36.976 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:01:59.618 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:37.991 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:38.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.091+0000 7fc8435d6640 10 monclient: get_monmap_and_config 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.091+0000 7fc8435d6640 10 monclient: build_initial_monmap 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.091+0000 7fc8435d6640 10 monclient: monmap: 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:43:38.092327+0000 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:43:38.092327+0000 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:43:38.091 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:43:38.092 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:38.093 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.092+0000 7fc8435d6640 1 Processor -- start 2024-03-25T19:43:38.093 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.092+0000 7fc8435d6640 1 -- start start 2024-03-25T19:43:38.093 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.092+0000 7fc8435d6640 10 monclient: init 2024-03-25T19:43:38.093 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.093+0000 7fc8435d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:38.093 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.093+0000 7fc8435d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:38.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.093+0000 7fc8435d6640 10 monclient(hunting): picked mon.noname-c con 0x7fc83c0b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:43:38.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.093+0000 7fc8435d6640 10 monclient(hunting): picked mon.noname-b con 0x7fc83c0b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:43:38.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.093+0000 7fc8435d6640 10 monclient(hunting): picked mon.noname-a con 0x7fc83c14be90 addr v1:172.21.15.81:6789/0 2024-03-25T19:43:38.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.093+0000 7fc8435d6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c0af940 con 0x7fc83c14be90 2024-03-25T19:43:38.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.093+0000 7fc8435d6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c0b0a70 con 0x7fc83c0b4d60 2024-03-25T19:43:38.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.093+0000 7fc8435d6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c0b0d50 con 0x7fc83c0b1730 2024-03-25T19:43:38.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.093+0000 7fc8435d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:38.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.093+0000 7fc8435d6640 10 monclient(hunting): authenticate will time out at 4053.175463s 2024-03-25T19:43:38.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.096+0000 7fc841372640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc83c0b4d60 0x7fc83c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:41600/0 (socket says 172.21.15.81:41600) 2024-03-25T19:43:38.096 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.096+0000 7fc841372640 1 -- 172.21.15.81:0/2403399370 learned_addr learned my addr 172.21.15.81:0/2403399370 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:43:38.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.107+0000 7fc833fff640 1 -- 172.21.15.81:0/2403399370 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930992491 0 0) 0x7fc83c0af940 con 0x7fc83c14be90 2024-03-25T19:43:38.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.107+0000 7fc833fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:38.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.107+0000 7fc833fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:38.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.107+0000 7fc833fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:38.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.107+0000 7fc833fff640 10 monclient(hunting): my global_id is 4296 2024-03-25T19:43:38.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.107+0000 7fc833fff640 1 -- 172.21.15.81:0/2403399370 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc828003520 con 0x7fc83c14be90 2024-03-25T19:43:38.118 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.119+0000 7fc833fff640 1 -- 172.21.15.81:0/2403399370 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4018104728 0 0) 0x7fc828003520 con 0x7fc83c14be90 2024-03-25T19:43:38.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.119+0000 7fc833fff640 1 -- 172.21.15.81:0/2403399370 >> v1:172.21.15.81:6790/0 conn(0x7fc83c0b1730 legacy=0x7fc83c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.119+0000 7fc833fff640 1 -- 172.21.15.81:0/2403399370 >> v1:172.21.15.98:6789/0 conn(0x7fc83c0b4d60 legacy=0x7fc83c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.119+0000 7fc833fff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:38.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.119+0000 7fc833fff640 1 monclient: found mon.noname-a 2024-03-25T19:43:38.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.119+0000 7fc833fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:38.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.119+0000 7fc833fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:43:38.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.119+0000 7fc833fff640 1 -- 172.21.15.81:0/2403399370 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83c0b1030 con 0x7fc83c14be90 2024-03-25T19:43:38.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.119+0000 7fc833fff640 10 monclient: _finish_auth 0 2024-03-25T19:43:38.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.119+0000 7fc833fff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:38.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.119+0000 7fc833fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:38.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.119+0000 7fc8435d6640 5 monclient: authenticate success, global_id 4296 2024-03-25T19:43:38.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.119+0000 7fc833fff640 1 -- 172.21.15.81:0/2403399370 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fc834001a10 con 0x7fc83c14be90 2024-03-25T19:43:38.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.119+0000 7fc833fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.120 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.119+0000 7fc833fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:38.120 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.119+0000 7fc833fff640 10 monclient: dump: 2024-03-25T19:43:38.120 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:38.120 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:38.120 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:38.120 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:38.120 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.120 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:38.120 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:38.120 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:38.120 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:38.120 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:38.120 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.119+0000 7fc8435d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:38.122 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.123+0000 7fc833fff640 1 -- 172.21.15.81:0/2403399370 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc834001b70 con 0x7fc83c14be90 2024-03-25T19:43:38.122 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.123+0000 7fc833fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:38.123 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.123+0000 7fc833fff640 1 -- 172.21.15.81:0/2403399370 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fc834001f80 con 0x7fc83c14be90 2024-03-25T19:43:38.123 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.123+0000 7fc8435d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:38.123 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.123+0000 7fc8435d6640 4 set_mon_vals no callback set 2024-03-25T19:43:38.123 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.123+0000 7fc833fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.123 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.123+0000 7fc833fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:38.123 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.123+0000 7fc833fff640 10 monclient: dump: 2024-03-25T19:43:38.123 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:38.123 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:38.123 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:38.123 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:38.123 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.123 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:38.124 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:38.124 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:38.124 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:38.124 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:38.126 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.124+0000 7fc8435d6640 10 monclient: shutdown 2024-03-25T19:43:38.126 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.124+0000 7fc8435d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:38.127 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.124+0000 7fc8435d6640 1 -- 172.21.15.81:0/2403399370 >> v1:172.21.15.81:6789/0 conn(0x7fc83c14be90 legacy=0x7fc83c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.127 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.124+0000 7fc8435d6640 1 -- 172.21.15.81:0/2403399370 shutdown_connections 2024-03-25T19:43:38.127 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.124+0000 7fc8435d6640 1 -- 172.21.15.81:0/2403399370 >> 172.21.15.81:0/2403399370 conn(0x7fc83c0617a0 msgr2=0x7fc83c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:38.127 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.124+0000 7fc8435d6640 1 -- 172.21.15.81:0/2403399370 shutdown_connections 2024-03-25T19:43:38.127 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.124+0000 7fc8435d6640 1 -- 172.21.15.81:0/2403399370 wait complete. 2024-03-25T19:43:38.127 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.124+0000 7fc8435d6640 10 monclient: build_initial_monmap 2024-03-25T19:43:38.127 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.124+0000 7fc8435d6640 10 monclient: monmap: 2024-03-25T19:43:38.127 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:43:38.127 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:38.127 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:43:38.127 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:43:38.127 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:38.127 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:43:38.128 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:43:38.128 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:43:38.128 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:43:38.128 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:38.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.125+0000 7fc8435d6640 1 librados: starting msgr at 2024-03-25T19:43:38.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.125+0000 7fc8435d6640 1 librados: starting objecter 2024-03-25T19:43:38.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.125+0000 7fc8435d6640 1 Processor -- start 2024-03-25T19:43:38.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.125+0000 7fc8435d6640 1 -- start start 2024-03-25T19:43:38.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.125+0000 7fc8435d6640 1 librados: setting wanted keys 2024-03-25T19:43:38.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.125+0000 7fc8435d6640 1 librados: calling monclient init 2024-03-25T19:43:38.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.125+0000 7fc8435d6640 10 monclient: init 2024-03-25T19:43:38.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.125+0000 7fc8435d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:38.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.125+0000 7fc8435d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:38.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.125+0000 7fc8435d6640 10 monclient(hunting): picked mon.noname-a con 0x7fc83c0b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:43:38.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.125+0000 7fc8435d6640 10 monclient(hunting): picked mon.noname-c con 0x7fc83c0b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:43:38.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.125+0000 7fc8435d6640 10 monclient(hunting): picked mon.noname-b con 0x7fc83c14be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:43:38.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.125+0000 7fc8435d6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c138150 con 0x7fc83c0b1730 2024-03-25T19:43:38.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.125+0000 7fc8435d6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c137c70 con 0x7fc83c14be90 2024-03-25T19:43:38.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.125+0000 7fc8435d6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83c137e00 con 0x7fc83c0b4d60 2024-03-25T19:43:38.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.125+0000 7fc8435d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:38.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.125+0000 7fc8435d6640 10 monclient(hunting): authenticate will time out at 4053.207744s 2024-03-25T19:43:38.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.134+0000 7fc840b71640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fc83c0b4d60 0x7fc83c1d7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:37936/0 (socket says 172.21.15.81:37936) 2024-03-25T19:43:38.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.134+0000 7fc840b71640 1 -- 172.21.15.81:0/140389413 learned_addr learned my addr 172.21.15.81:0/140389413 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:43:38.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.135+0000 7fc831ffb640 10 client.?.objecter ms_handle_connect 0x7fc83c0b4d60 2024-03-25T19:43:38.135 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.135+0000 7fc831ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:38.139 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.140+0000 7fc831ffb640 10 client.?.objecter ms_handle_connect 0x7fc83c0b1730 2024-03-25T19:43:38.139 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.140+0000 7fc831ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:38.145 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.145+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323301629 0 0) 0x7fc83c137e00 con 0x7fc83c0b4d60 2024-03-25T19:43:38.145 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.145+0000 7fc831ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:38.145 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.145+0000 7fc831ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:38.145 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.145+0000 7fc831ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:38.145 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.145+0000 7fc831ffb640 10 monclient(hunting): my global_id is 4283 2024-03-25T19:43:38.145 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.145+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc820004eb0 con 0x7fc83c0b4d60 2024-03-25T19:43:38.151 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.152+0000 7fc831ffb640 10 client.?.objecter ms_handle_connect 0x7fc83c14be90 2024-03-25T19:43:38.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.152+0000 7fc831ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:38.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.153+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529262928 0 0) 0x7fc83c138150 con 0x7fc83c0b1730 2024-03-25T19:43:38.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.153+0000 7fc831ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:38.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.153+0000 7fc831ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:38.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.153+0000 7fc831ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:38.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.153+0000 7fc831ffb640 10 monclient(hunting): my global_id is 4299 2024-03-25T19:43:38.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.153+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc83c137e00 con 0x7fc83c0b1730 2024-03-25T19:43:38.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.157+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2395004755 0 0) 0x7fc820004eb0 con 0x7fc83c0b4d60 2024-03-25T19:43:38.157 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.157+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc83c138150 con 0x7fc83c0b4d60 2024-03-25T19:43:38.157 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.157+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fc82c001a30 con 0x7fc83c0b4d60 2024-03-25T19:43:38.157 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.157+0000 7fc831ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.157 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.157+0000 7fc831ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:38.157 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.157+0000 7fc831ffb640 10 monclient(hunting): dump: 2024-03-25T19:43:38.157 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:38.157 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:38.157 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:38.157 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:38.157 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.157 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:38.158 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:38.158 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:38.158 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:38.158 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:38.158 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.157+0000 7fc831ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:38.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.163+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685150586 0 0) 0x7fc83c137c70 con 0x7fc83c14be90 2024-03-25T19:43:38.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.163+0000 7fc831ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:38.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.163+0000 7fc831ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:38.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.163+0000 7fc831ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:38.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.163+0000 7fc831ffb640 10 monclient(hunting): my global_id is 4297 2024-03-25T19:43:38.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.163+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc820004eb0 con 0x7fc83c14be90 2024-03-25T19:43:38.168 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4254599676 0 0) 0x7fc83c138150 con 0x7fc83c0b4d60 2024-03-25T19:43:38.168 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 >> v1:172.21.15.98:6789/0 conn(0x7fc83c14be90 legacy=0x7fc83c1d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 >> v1:172.21.15.81:6789/0 conn(0x7fc83c0b1730 legacy=0x7fc83c131da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc831ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:43:38.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc831ffb640 1 monclient: found mon.c 2024-03-25T19:43:38.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc831ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:38.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc831ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:43:38.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83c1386a0 con 0x7fc83c0b4d60 2024-03-25T19:43:38.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc831ffb640 10 monclient: _finish_auth 0 2024-03-25T19:43:38.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc831ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:43:38.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc831ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:38.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc8435d6640 5 monclient: authenticate success, global_id 4283 2024-03-25T19:43:38.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc8435d6640 10 monclient: _renew_subs 2024-03-25T19:43:38.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc8435d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:43:38.170 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc8435d6640 1 -- 172.21.15.81:0/140389413 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc83c144e00 con 0x7fc83c0b4d60 2024-03-25T19:43:38.170 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc8435d6640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:38.170 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc8435d6640 10 monclient: _renew_subs 2024-03-25T19:43:38.170 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc8435d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:43:38.170 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc8435d6640 1 -- 172.21.15.81:0/140389413 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc83c132690 con 0x7fc83c0b4d60 2024-03-25T19:43:38.170 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.168+0000 7fc8435d6640 1 librados: init done 2024-03-25T19:43:38.171 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.169+0000 7fc8435d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:38.171 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.169+0000 7fc8435d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:38.171 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.169+0000 7fc8435d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:43:38.171 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.169+0000 7fc8435d6640 1 -- 172.21.15.81:0/140389413 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc83c1451a0 con 0x7fc83c0b4d60 2024-03-25T19:43:38.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.486+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc82c001c70 con 0x7fc83c0b4d60 2024-03-25T19:43:38.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.486+0000 7fc831ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:38.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.486+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fc82c003a60 con 0x7fc83c0b4d60 2024-03-25T19:43:38.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.486+0000 7fc831ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:38.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.486+0000 7fc831ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:38.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.486+0000 7fc831ffb640 10 monclient: dump: 2024-03-25T19:43:38.486 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:38.486 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:38.486 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:38.486 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:38.486 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:38.486 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:38.487 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:38.487 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:38.487 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:38.487 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:38.487 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.486+0000 7fc8337fe640 4 set_mon_vals no callback set 2024-03-25T19:43:38.487 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.487+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fc82c003c40 con 0x7fc83c0b4d60 2024-03-25T19:43:38.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.488+0000 7fc841372640 10 monclient: get_auth_request con 0x7fc82006de00 auth_method 0 2024-03-25T19:43:38.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.488+0000 7fc831ffb640 10 client.4283.objecter ms_handle_connect 0x7fc82006de00 2024-03-25T19:43:38.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.493+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fc82c054950 con 0x7fc83c0b4d60 2024-03-25T19:43:38.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.493+0000 7fc831ffb640 10 client.4283.objecter ms_dispatch 0x7fc83c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:38.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.493+0000 7fc831ffb640 3 client.4283.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:38.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.493+0000 7fc831ffb640 3 client.4283.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:38.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.493+0000 7fc831ffb640 20 client.4283.objecter dump_active .. 0 homeless 2024-03-25T19:43:38.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.493+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc82c050460 con 0x7fc83c0b4d60 2024-03-25T19:43:38.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.493+0000 7fc831ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:38.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.493+0000 7fc831ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:38.739 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.740+0000 7fc8435d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:38.740 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.740+0000 7fc8435d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:38.740 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.740+0000 7fc8435d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:43:38.740 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.740+0000 7fc8435d6640 1 -- 172.21.15.81:0/140389413 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc83c0627b0 con 0x7fc83c0b4d60 2024-03-25T19:43:38.759 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.759+0000 7fc831ffb640 1 -- 172.21.15.81:0/140389413 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fc82c052390 con 0x7fc83c0b4d60 2024-03-25T19:43:38.759 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.759+0000 7fc831ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:38.759 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.759+0000 7fc831ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:38.759 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:43:38.759 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:38.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.761+0000 7fc8435d6640 10 librados: watch_flush enter 2024-03-25T19:43:38.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.761+0000 7fc8435d6640 10 librados: watch_flush exit 2024-03-25T19:43:38.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.761+0000 7fc8435d6640 20 client.4283.objecter shutdown clearing up homeless session... 2024-03-25T19:43:38.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.761+0000 7fc8435d6640 10 client.4283.objecter successfully canceled tick 2024-03-25T19:43:38.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.762+0000 7fc8435d6640 1 -- 172.21.15.81:0/140389413 >> v1:172.21.15.98:6800/3980829775 conn(0x7fc82006de00 legacy=0x7fc8200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.762+0000 7fc8435d6640 10 monclient: shutdown 2024-03-25T19:43:38.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.762+0000 7fc8435d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:38.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.762+0000 7fc8435d6640 1 -- 172.21.15.81:0/140389413 >> v1:172.21.15.81:6790/0 conn(0x7fc83c0b4d60 legacy=0x7fc83c1d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:38.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.762+0000 7fc8435d6640 1 -- 172.21.15.81:0/140389413 shutdown_connections 2024-03-25T19:43:38.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.762+0000 7fc8435d6640 1 -- 172.21.15.81:0/140389413 >> 172.21.15.81:0/140389413 conn(0x7fc83c0617a0 msgr2=0x7fc83c0b6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:38.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.762+0000 7fc8435d6640 1 -- 172.21.15.81:0/140389413 shutdown_connections 2024-03-25T19:43:38.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.762+0000 7fc8435d6640 1 -- 172.21.15.81:0/140389413 wait complete. 2024-03-25T19:43:38.763 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:38.762+0000 7fc8435d6640 1 librados: shutdown 2024-03-25T19:43:38.773 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:45.579 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:45.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.694+0000 7f492a44a640 10 monclient: get_monmap_and_config 2024-03-25T19:43:45.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.694+0000 7f492a44a640 10 monclient: build_initial_monmap 2024-03-25T19:43:45.694 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.694+0000 7f492a44a640 10 monclient: monmap: 2024-03-25T19:43:45.694 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:43:45.694 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:45.694 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:43:45.695115+0000 2024-03-25T19:43:45.694 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:43:45.695115+0000 2024-03-25T19:43:45.694 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:45.694 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:45.694 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:43:45.694 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:43:45.694 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:43:45.694 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:45.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.695+0000 7f492a44a640 1 Processor -- start 2024-03-25T19:43:45.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.695+0000 7f492a44a640 1 -- start start 2024-03-25T19:43:45.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.695+0000 7f492a44a640 10 monclient: init 2024-03-25T19:43:45.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.695+0000 7f492a44a640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:45.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.695+0000 7f492a44a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:45.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.695+0000 7f492a44a640 10 monclient(hunting): picked mon.noname-a con 0x7f49240705b0 addr v1:172.21.15.81:6789/0 2024-03-25T19:43:45.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.695+0000 7f492a44a640 10 monclient(hunting): picked mon.noname-b con 0x7f49241402d0 addr v1:172.21.15.81:6790/0 2024-03-25T19:43:45.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.695+0000 7f492a44a640 10 monclient(hunting): picked mon.noname-c con 0x7f4924149ae0 addr v1:172.21.15.98:6789/0 2024-03-25T19:43:45.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.695+0000 7f492a44a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49240718c0 con 0x7f49240705b0 2024-03-25T19:43:45.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.696+0000 7f492a44a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49240714c0 con 0x7f4924149ae0 2024-03-25T19:43:45.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.696+0000 7f492a44a640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4924072b50 con 0x7f49241402d0 2024-03-25T19:43:45.697 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.696+0000 7f492a44a640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:45.697 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.696+0000 7f492a44a640 10 monclient(hunting): authenticate will time out at 4060.778028s 2024-03-25T19:43:45.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.700+0000 7f49289e7640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4924149ae0 0x7f492414bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:46532/0 (socket says 172.21.15.81:46532) 2024-03-25T19:43:45.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.700+0000 7f49289e7640 1 -- 172.21.15.81:0/1361240214 learned_addr learned my addr 172.21.15.81:0/1361240214 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:43:45.715 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.716+0000 7f4922ffd640 1 -- 172.21.15.81:0/1361240214 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121222620 0 0) 0x7f49240714c0 con 0x7f4924149ae0 2024-03-25T19:43:45.715 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.716+0000 7f4922ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:45.715 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.716+0000 7f4922ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:45.716 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.716+0000 7f4922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:45.716 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.716+0000 7f4922ffd640 10 monclient(hunting): my global_id is 4300 2024-03-25T19:43:45.716 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.716+0000 7f4922ffd640 1 -- 172.21.15.81:0/1361240214 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f490c003520 con 0x7f4924149ae0 2024-03-25T19:43:45.721 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.722+0000 7f4922ffd640 1 -- 172.21.15.81:0/1361240214 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503294112 0 0) 0x7f4924072b50 con 0x7f49241402d0 2024-03-25T19:43:45.721 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.722+0000 7f4922ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:45.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.722+0000 7f4922ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:45.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.722+0000 7f4922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:45.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.722+0000 7f4922ffd640 10 monclient(hunting): my global_id is 4286 2024-03-25T19:43:45.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.722+0000 7f4922ffd640 1 -- 172.21.15.81:0/1361240214 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49240714c0 con 0x7f49241402d0 2024-03-25T19:43:45.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.722+0000 7f4922ffd640 1 -- 172.21.15.81:0/1361240214 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552688925 0 0) 0x7f49240718c0 con 0x7f49240705b0 2024-03-25T19:43:45.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.722+0000 7f4922ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:45.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.722+0000 7f4922ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:45.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.722+0000 7f4922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:45.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.722+0000 7f4922ffd640 10 monclient(hunting): my global_id is 4302 2024-03-25T19:43:45.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.722+0000 7f4922ffd640 1 -- 172.21.15.81:0/1361240214 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4924072b50 con 0x7f49240705b0 2024-03-25T19:43:45.724 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.724+0000 7f4922ffd640 1 -- 172.21.15.81:0/1361240214 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1928516920 0 0) 0x7f490c003520 con 0x7f4924149ae0 2024-03-25T19:43:45.724 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.724+0000 7f4922ffd640 1 -- 172.21.15.81:0/1361240214 >> v1:172.21.15.81:6790/0 conn(0x7f49241402d0 legacy=0x7f4924140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:45.724 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.724+0000 7f4922ffd640 1 -- 172.21.15.81:0/1361240214 >> v1:172.21.15.81:6789/0 conn(0x7f49240705b0 legacy=0x7f492413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:45.724 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.725+0000 7f4922ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:45.724 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.725+0000 7f4922ffd640 1 monclient: found mon.noname-c 2024-03-25T19:43:45.725 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.725+0000 7f4922ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:45.725 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.725+0000 7f4922ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:43:45.725 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.725+0000 7f4922ffd640 1 -- 172.21.15.81:0/1361240214 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492413dd10 con 0x7f4924149ae0 2024-03-25T19:43:45.725 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.725+0000 7f4922ffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:45.725 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.725+0000 7f4922ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:45.725 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.725+0000 7f4922ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:45.725 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.725+0000 7f4922ffd640 1 -- 172.21.15.81:0/1361240214 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f491c001a10 con 0x7f4924149ae0 2024-03-25T19:43:45.725 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.725+0000 7f4922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:45.725 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.725+0000 7f4922ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:45.725 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.725+0000 7f4922ffd640 10 monclient: dump: 2024-03-25T19:43:45.725 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:45.725 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:45.725 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:45.725 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:45.726 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:45.726 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:45.726 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:45.726 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:45.726 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:45.726 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:45.726 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.725+0000 7f492a44a640 5 monclient: authenticate success, global_id 4300 2024-03-25T19:43:45.726 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.725+0000 7f492a44a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:45.734 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.735+0000 7f4922ffd640 1 -- 172.21.15.81:0/1361240214 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f491c001b70 con 0x7f4924149ae0 2024-03-25T19:43:45.734 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.735+0000 7f4922ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:45.734 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.735+0000 7f4922ffd640 1 -- 172.21.15.81:0/1361240214 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f491c001f80 con 0x7f4924149ae0 2024-03-25T19:43:45.735 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.735+0000 7f4922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:45.735 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.735+0000 7f4922ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:45.735 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.735+0000 7f4922ffd640 10 monclient: dump: 2024-03-25T19:43:45.735 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:45.735 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:45.735 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:45.735 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:45.735 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:45.735 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:45.735 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:45.735 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:45.735 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:45.735 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:45.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.735+0000 7f492a44a640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:45.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.735+0000 7f492a44a640 4 set_mon_vals no callback set 2024-03-25T19:43:45.737 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.736+0000 7f492a44a640 10 monclient: shutdown 2024-03-25T19:43:45.737 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.736+0000 7f492a44a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:45.737 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.736+0000 7f492a44a640 1 -- 172.21.15.81:0/1361240214 >> v1:172.21.15.98:6789/0 conn(0x7f4924149ae0 legacy=0x7f492414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:45.737 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.736+0000 7f492a44a640 1 -- 172.21.15.81:0/1361240214 shutdown_connections 2024-03-25T19:43:45.737 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.736+0000 7f492a44a640 1 -- 172.21.15.81:0/1361240214 >> 172.21.15.81:0/1361240214 conn(0x7f49240b5d10 msgr2=0x7f49240b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:45.737 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.736+0000 7f492a44a640 1 -- 172.21.15.81:0/1361240214 shutdown_connections 2024-03-25T19:43:45.737 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.736+0000 7f492a44a640 1 -- 172.21.15.81:0/1361240214 wait complete. 2024-03-25T19:43:45.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.737+0000 7f492a44a640 10 monclient: build_initial_monmap 2024-03-25T19:43:45.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.737+0000 7f492a44a640 10 monclient: monmap: 2024-03-25T19:43:45.738 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:43:45.738 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:45.738 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:43:45.738 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:43:45.738 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:45.738 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:43:45.738 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:43:45.738 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:43:45.738 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:43:45.738 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:45.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.737+0000 7f492a44a640 1 librados: starting msgr at 2024-03-25T19:43:45.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.737+0000 7f492a44a640 1 librados: starting objecter 2024-03-25T19:43:45.739 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.737+0000 7f492a44a640 1 Processor -- start 2024-03-25T19:43:45.739 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.738+0000 7f492a44a640 1 -- start start 2024-03-25T19:43:45.739 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.738+0000 7f492a44a640 1 librados: setting wanted keys 2024-03-25T19:43:45.739 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.738+0000 7f492a44a640 1 librados: calling monclient init 2024-03-25T19:43:45.739 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.738+0000 7f492a44a640 10 monclient: init 2024-03-25T19:43:45.740 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.738+0000 7f492a44a640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:45.740 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.738+0000 7f492a44a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:45.740 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.739+0000 7f492a44a640 10 monclient(hunting): picked mon.noname-a con 0x7f49240705b0 addr v1:172.21.15.81:6789/0 2024-03-25T19:43:45.740 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.739+0000 7f492a44a640 10 monclient(hunting): picked mon.noname-b con 0x7f49241402d0 addr v1:172.21.15.98:6789/0 2024-03-25T19:43:45.740 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.739+0000 7f492a44a640 10 monclient(hunting): picked mon.noname-c con 0x7f4924149ae0 addr v1:172.21.15.81:6790/0 2024-03-25T19:43:45.740 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.739+0000 7f492a44a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4924137370 con 0x7f49240705b0 2024-03-25T19:43:45.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.739+0000 7f492a44a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4924131d90 con 0x7f49241402d0 2024-03-25T19:43:45.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.739+0000 7f492a44a640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4924143700 con 0x7f4924149ae0 2024-03-25T19:43:45.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.739+0000 7f492a44a640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:45.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.739+0000 7f492a44a640 10 monclient(hunting): authenticate will time out at 4060.821434s 2024-03-25T19:43:45.743 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.743+0000 7f4923fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f49240705b0 0x7f4924141de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:55286/0 (socket says 172.21.15.81:55286) 2024-03-25T19:43:45.743 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.743+0000 7f4923fff640 1 -- 172.21.15.81:0/2566390550 learned_addr learned my addr 172.21.15.81:0/2566390550 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:43:45.745 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.745+0000 7f4920ff9640 10 client.?.objecter ms_handle_connect 0x7f49240705b0 2024-03-25T19:43:45.745 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.745+0000 7f4920ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:45.752 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.753+0000 7f4920ff9640 10 client.?.objecter ms_handle_connect 0x7f4924149ae0 2024-03-25T19:43:45.753 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.753+0000 7f4920ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:45.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.757+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070734960 0 0) 0x7f4924137370 con 0x7f49240705b0 2024-03-25T19:43:45.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.757+0000 7f4920ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:45.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.757+0000 7f4920ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:45.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.757+0000 7f4920ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:45.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.757+0000 7f4920ff9640 10 monclient(hunting): my global_id is 4305 2024-03-25T19:43:45.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.757+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4910004eb0 con 0x7f49240705b0 2024-03-25T19:43:45.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.763+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3535230351 0 0) 0x7f4910004eb0 con 0x7f49240705b0 2024-03-25T19:43:45.763 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.763+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4924137370 con 0x7f49240705b0 2024-03-25T19:43:45.763 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.763+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4914001a10 con 0x7f49240705b0 2024-03-25T19:43:45.763 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.763+0000 7f4920ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:45.763 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.763+0000 7f4920ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:45.763 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.763+0000 7f4920ff9640 10 monclient(hunting): dump: 2024-03-25T19:43:45.763 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:45.763 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:45.763 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:45.763 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:45.763 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:45.763 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:45.763 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:45.763 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:45.764 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:45.764 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:45.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.763+0000 7f4920ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:45.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.764+0000 7f4920ff9640 10 client.?.objecter ms_handle_connect 0x7f49241402d0 2024-03-25T19:43:45.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.764+0000 7f4920ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:45.773 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.774+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474822253 0 0) 0x7f4924131d90 con 0x7f49241402d0 2024-03-25T19:43:45.773 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.774+0000 7f4920ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:45.774 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.774+0000 7f4920ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:45.774 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.774+0000 7f4920ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:45.774 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.774+0000 7f4920ff9640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:43:45.774 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.774+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4910004eb0 con 0x7f49241402d0 2024-03-25T19:43:45.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.778+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530619104 0 0) 0x7f4924143700 con 0x7f4924149ae0 2024-03-25T19:43:45.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.778+0000 7f4920ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:43:45.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.778+0000 7f4920ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:45.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.778+0000 7f4920ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:45.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.778+0000 7f4920ff9640 10 monclient(hunting): my global_id is 4289 2024-03-25T19:43:45.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.778+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4924131d90 con 0x7f4924149ae0 2024-03-25T19:43:45.780 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.780+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1505121854 0 0) 0x7f4924137370 con 0x7f49240705b0 2024-03-25T19:43:45.780 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.780+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 >> v1:172.21.15.81:6790/0 conn(0x7f4924149ae0 legacy=0x7f4924131680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:45.780 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.780+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 >> v1:172.21.15.98:6789/0 conn(0x7f49241402d0 legacy=0x7f4924130f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:45.781 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.780+0000 7f4920ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:43:45.781 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.780+0000 7f4920ff9640 1 monclient: found mon.a 2024-03-25T19:43:45.781 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.780+0000 7f4920ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:45.781 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.780+0000 7f4920ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:43:45.781 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.780+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49241439e0 con 0x7f49240705b0 2024-03-25T19:43:45.781 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.780+0000 7f4920ff9640 10 monclient: _finish_auth 0 2024-03-25T19:43:45.781 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.780+0000 7f4920ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:43:45.781 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.780+0000 7f4920ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:45.781 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.781+0000 7f492a44a640 5 monclient: authenticate success, global_id 4305 2024-03-25T19:43:45.781 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.781+0000 7f492a44a640 10 monclient: _renew_subs 2024-03-25T19:43:45.781 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.781+0000 7f492a44a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:43:45.781 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.781+0000 7f492a44a640 1 -- 172.21.15.81:0/2566390550 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49241418c0 con 0x7f49240705b0 2024-03-25T19:43:45.781 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.781+0000 7f492a44a640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:45.781 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.781+0000 7f492a44a640 10 monclient: _renew_subs 2024-03-25T19:43:45.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.781+0000 7f492a44a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:43:45.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.781+0000 7f492a44a640 1 -- 172.21.15.81:0/2566390550 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49241443c0 con 0x7f49240705b0 2024-03-25T19:43:45.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.781+0000 7f492a44a640 1 librados: init done 2024-03-25T19:43:45.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.782+0000 7f492a44a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:45.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.782+0000 7f492a44a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:45.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.782+0000 7f492a44a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:43:45.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:45.782+0000 7f492a44a640 1 -- 172.21.15.81:0/2566390550 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4924141730 con 0x7f49240705b0 2024-03-25T19:43:46.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.090+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4914001cf0 con 0x7f49240705b0 2024-03-25T19:43:46.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.090+0000 7f4920ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:46.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.090+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4914003a60 con 0x7f49240705b0 2024-03-25T19:43:46.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.090+0000 7f4920ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:46.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.090+0000 7f4920ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:46.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.090+0000 7f4920ff9640 10 monclient: dump: 2024-03-25T19:43:46.091 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:46.091 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:46.091 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:46.091 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:46.091 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:46.091 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:46.091 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:46.091 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:46.091 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:46.092 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:46.092 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.090+0000 7f49227fc640 4 set_mon_vals no callback set 2024-03-25T19:43:46.092 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.091+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f4914003c80 con 0x7f49240705b0 2024-03-25T19:43:46.092 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.091+0000 7f49237fe640 10 monclient: get_auth_request con 0x7f491006de50 auth_method 0 2024-03-25T19:43:46.092 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.092+0000 7f4920ff9640 10 client.4305.objecter ms_handle_connect 0x7f491006de50 2024-03-25T19:43:46.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.097+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f49140548c0 con 0x7f49240705b0 2024-03-25T19:43:46.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.097+0000 7f4920ff9640 10 client.4305.objecter ms_dispatch 0x7f4924000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:46.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.097+0000 7f4920ff9640 3 client.4305.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:46.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.097+0000 7f4920ff9640 3 client.4305.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:46.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.097+0000 7f4920ff9640 20 client.4305.objecter dump_active .. 0 homeless 2024-03-25T19:43:46.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.097+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49140503d0 con 0x7f49240705b0 2024-03-25T19:43:46.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.097+0000 7f4920ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:46.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.097+0000 7f4920ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:46.347 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.347+0000 7f492a44a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:46.347 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.347+0000 7f492a44a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:46.347 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.347+0000 7f492a44a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:43:46.347 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.347+0000 7f492a44a640 1 -- 172.21.15.81:0/2566390550 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4924141a50 con 0x7f49240705b0 2024-03-25T19:43:46.378 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.378+0000 7f4920ff9640 1 -- 172.21.15.81:0/2566390550 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f4914052300 con 0x7f49240705b0 2024-03-25T19:43:46.378 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.378+0000 7f4920ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:46.378 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.378+0000 7f4920ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:46.378 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:43:46.379 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:46.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.381+0000 7f492a44a640 10 librados: watch_flush enter 2024-03-25T19:43:46.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.381+0000 7f492a44a640 10 librados: watch_flush exit 2024-03-25T19:43:46.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.381+0000 7f492a44a640 20 client.4305.objecter shutdown clearing up homeless session... 2024-03-25T19:43:46.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.381+0000 7f492a44a640 10 client.4305.objecter successfully canceled tick 2024-03-25T19:43:46.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.381+0000 7f492a44a640 1 -- 172.21.15.81:0/2566390550 >> v1:172.21.15.98:6800/3980829775 conn(0x7f491006de50 legacy=0x7f4910070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.381+0000 7f492a44a640 10 monclient: shutdown 2024-03-25T19:43:46.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.381+0000 7f492a44a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:46.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.381+0000 7f492a44a640 1 -- 172.21.15.81:0/2566390550 >> v1:172.21.15.81:6789/0 conn(0x7f49240705b0 legacy=0x7f4924141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:46.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.381+0000 7f492a44a640 1 -- 172.21.15.81:0/2566390550 shutdown_connections 2024-03-25T19:43:46.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.381+0000 7f492a44a640 1 -- 172.21.15.81:0/2566390550 >> 172.21.15.81:0/2566390550 conn(0x7f49240b5d10 msgr2=0x7f4924140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:46.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.381+0000 7f492a44a640 1 -- 172.21.15.81:0/2566390550 shutdown_connections 2024-03-25T19:43:46.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.381+0000 7f492a44a640 1 -- 172.21.15.81:0/2566390550 wait complete. 2024-03-25T19:43:46.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:46.381+0000 7f492a44a640 1 librados: shutdown 2024-03-25T19:43:46.394 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:53.200 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:53.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.314+0000 7fa844a25640 10 monclient: get_monmap_and_config 2024-03-25T19:43:53.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.314+0000 7fa844a25640 10 monclient: build_initial_monmap 2024-03-25T19:43:53.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.314+0000 7fa844a25640 10 monclient: monmap: 2024-03-25T19:43:53.314 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:43:53.314 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:53.314 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:43:53.315773+0000 2024-03-25T19:43:53.314 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:43:53.315773+0000 2024-03-25T19:43:53.315 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:53.315 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:53.315 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:43:53.315 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:43:53.315 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:43:53.315 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.315+0000 7fa844a25640 1 Processor -- start 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.315+0000 7fa844a25640 1 -- start start 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.315+0000 7fa844a25640 10 monclient: init 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.316+0000 7fa844a25640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.316+0000 7fa844a25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.316+0000 7fa844a25640 10 monclient(hunting): picked mon.noname-b con 0x7fa8400a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.316+0000 7fa844a25640 10 monclient(hunting): picked mon.noname-c con 0x7fa84013f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.316+0000 7fa844a25640 10 monclient(hunting): picked mon.noname-a con 0x7fa8401430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:43:53.316 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.316+0000 7fa844a25640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa840142520 con 0x7fa8401430c0 2024-03-25T19:43:53.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.316+0000 7fa844a25640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8400b28a0 con 0x7fa84013f710 2024-03-25T19:43:53.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.316+0000 7fa844a25640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8400b2a30 con 0x7fa8400a98f0 2024-03-25T19:43:53.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.316+0000 7fa844a25640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:53.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.316+0000 7fa844a25640 10 monclient(hunting): authenticate will time out at 4068.398694s 2024-03-25T19:43:53.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.318+0000 7fa83ed9d640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fa8401430c0 0x7fa84014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:51056/0 (socket says 172.21.15.81:51056) 2024-03-25T19:43:53.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.318+0000 7fa83ed9d640 1 -- 172.21.15.81:0/1118850439 learned_addr learned my addr 172.21.15.81:0/1118850439 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:43:53.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.336+0000 7fa83dd9b640 1 -- 172.21.15.81:0/1118850439 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137628149 0 0) 0x7fa840142520 con 0x7fa8401430c0 2024-03-25T19:43:53.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.336+0000 7fa83dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.336+0000 7fa83dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.336+0000 7fa83dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.336+0000 7fa83dd9b640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:43:53.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.336+0000 7fa83dd9b640 1 -- 172.21.15.81:0/1118850439 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa82c003520 con 0x7fa8401430c0 2024-03-25T19:43:53.345 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.345+0000 7fa83dd9b640 1 -- 172.21.15.81:0/1118850439 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639436185 0 0) 0x7fa8400b2a30 con 0x7fa8400a98f0 2024-03-25T19:43:53.345 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.345+0000 7fa83dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.345 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.345+0000 7fa83dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.345 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.345+0000 7fa83dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.345 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.345+0000 7fa83dd9b640 10 monclient(hunting): my global_id is 4292 2024-03-25T19:43:53.345 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.345+0000 7fa83dd9b640 1 -- 172.21.15.81:0/1118850439 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa840142520 con 0x7fa8400a98f0 2024-03-25T19:43:53.347 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.348+0000 7fa83dd9b640 1 -- 172.21.15.81:0/1118850439 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028511864 0 0) 0x7fa8400b28a0 con 0x7fa84013f710 2024-03-25T19:43:53.347 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.348+0000 7fa83dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.347 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.348+0000 7fa83dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.347 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.348+0000 7fa83dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.348 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.348+0000 7fa83dd9b640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:43:53.348 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.348+0000 7fa83dd9b640 1 -- 172.21.15.81:0/1118850439 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8400b2a30 con 0x7fa84013f710 2024-03-25T19:43:53.353 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.353+0000 7fa83dd9b640 1 -- 172.21.15.81:0/1118850439 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 866800125 0 0) 0x7fa82c003520 con 0x7fa8401430c0 2024-03-25T19:43:53.353 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.353+0000 7fa83dd9b640 1 -- 172.21.15.81:0/1118850439 >> v1:172.21.15.81:6790/0 conn(0x7fa8400a98f0 legacy=0x7fa8400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.353 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.353+0000 7fa83dd9b640 1 -- 172.21.15.81:0/1118850439 >> v1:172.21.15.98:6789/0 conn(0x7fa84013f710 legacy=0x7fa8401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.353 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.353+0000 7fa83dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:43:53.353 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.353+0000 7fa83dd9b640 1 monclient: found mon.noname-a 2024-03-25T19:43:53.353 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.353+0000 7fa83dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:53.353 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.353+0000 7fa83dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:43:53.353 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.353+0000 7fa83dd9b640 1 -- 172.21.15.81:0/1118850439 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8400b44c0 con 0x7fa8401430c0 2024-03-25T19:43:53.353 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.353+0000 7fa83dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:43:53.354 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.353+0000 7fa83dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:43:53.354 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.353+0000 7fa83dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:53.354 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.353+0000 7fa83dd9b640 1 -- 172.21.15.81:0/1118850439 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa838001a10 con 0x7fa8401430c0 2024-03-25T19:43:53.354 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.353+0000 7fa83dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.354 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.353+0000 7fa83dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:53.354 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.353+0000 7fa83dd9b640 10 monclient: dump: 2024-03-25T19:43:53.354 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:53.354 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:53.354 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:53.354 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:53.354 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.354 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:53.354 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:53.354 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:53.355 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:53.355 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:53.355 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.353+0000 7fa844a25640 5 monclient: authenticate success, global_id 4308 2024-03-25T19:43:53.355 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.353+0000 7fa844a25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:53.357 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.358+0000 7fa83dd9b640 1 -- 172.21.15.81:0/1118850439 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa838001b70 con 0x7fa8401430c0 2024-03-25T19:43:53.357 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.358+0000 7fa83dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:53.357 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.358+0000 7fa83dd9b640 1 -- 172.21.15.81:0/1118850439 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa838001f80 con 0x7fa8401430c0 2024-03-25T19:43:53.358 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.358+0000 7fa844a25640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:53.358 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.358+0000 7fa844a25640 4 set_mon_vals no callback set 2024-03-25T19:43:53.358 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.358+0000 7fa83dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.358 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.358+0000 7fa83dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:53.358 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.358+0000 7fa83dd9b640 10 monclient: dump: 2024-03-25T19:43:53.358 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:53.358 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:53.358 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:53.358 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:53.358 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.358 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:53.358 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:53.358 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:53.359 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:53.359 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:53.360 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.359+0000 7fa844a25640 10 monclient: shutdown 2024-03-25T19:43:53.360 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.359+0000 7fa844a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:53.360 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.359+0000 7fa844a25640 1 -- 172.21.15.81:0/1118850439 >> v1:172.21.15.81:6789/0 conn(0x7fa8401430c0 legacy=0x7fa84014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.360 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.359+0000 7fa844a25640 1 -- 172.21.15.81:0/1118850439 shutdown_connections 2024-03-25T19:43:53.360 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.359+0000 7fa844a25640 1 -- 172.21.15.81:0/1118850439 >> 172.21.15.81:0/1118850439 conn(0x7fa8400b7ea0 msgr2=0x7fa8400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:53.360 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.359+0000 7fa844a25640 1 -- 172.21.15.81:0/1118850439 shutdown_connections 2024-03-25T19:43:53.360 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.359+0000 7fa844a25640 1 -- 172.21.15.81:0/1118850439 wait complete. 2024-03-25T19:43:53.360 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.360+0000 7fa844a25640 10 monclient: build_initial_monmap 2024-03-25T19:43:53.360 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.360+0000 7fa844a25640 10 monclient: monmap: 2024-03-25T19:43:53.360 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:43:53.360 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.360+0000 7fa844a25640 1 librados: starting msgr at 2024-03-25T19:43:53.361 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.361+0000 7fa844a25640 1 librados: starting objecter 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.361+0000 7fa844a25640 1 Processor -- start 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.361+0000 7fa844a25640 1 -- start start 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.361+0000 7fa844a25640 1 librados: setting wanted keys 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.361+0000 7fa844a25640 1 librados: calling monclient init 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.361+0000 7fa844a25640 10 monclient: init 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.362+0000 7fa844a25640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.362+0000 7fa844a25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.362+0000 7fa844a25640 10 monclient(hunting): picked mon.noname-a con 0x7fa8400a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.362+0000 7fa844a25640 10 monclient(hunting): picked mon.noname-b con 0x7fa84013f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.362+0000 7fa844a25640 10 monclient(hunting): picked mon.noname-c con 0x7fa8401430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.362+0000 7fa844a25640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8401d7df0 con 0x7fa8400a98f0 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.362+0000 7fa844a25640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8401d7f80 con 0x7fa84013f710 2024-03-25T19:43:53.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.362+0000 7fa844a25640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa84013b5c0 con 0x7fa8401430c0 2024-03-25T19:43:53.364 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.362+0000 7fa844a25640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:53.364 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.362+0000 7fa844a25640 10 monclient(hunting): authenticate will time out at 4068.444137s 2024-03-25T19:43:53.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.369+0000 7fa83e59c640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fa8400a98f0 0x7fa8401d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:51098/0 (socket says 172.21.15.81:51098) 2024-03-25T19:43:53.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.369+0000 7fa83e59c640 1 -- 172.21.15.81:0/4064319290 learned_addr learned my addr 172.21.15.81:0/4064319290 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:43:53.372 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.372+0000 7fa836ffd640 10 client.?.objecter ms_handle_connect 0x7fa8400a98f0 2024-03-25T19:43:53.372 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.372+0000 7fa836ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:53.374 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.374+0000 7fa836ffd640 10 client.?.objecter ms_handle_connect 0x7fa8401430c0 2024-03-25T19:43:53.374 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.374+0000 7fa836ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:53.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.383+0000 7fa836ffd640 10 client.?.objecter ms_handle_connect 0x7fa84013f710 2024-03-25T19:43:53.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.383+0000 7fa836ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:53.384 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.385+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284801452 0 0) 0x7fa8401d7df0 con 0x7fa8400a98f0 2024-03-25T19:43:53.384 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.385+0000 7fa836ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.384 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.385+0000 7fa836ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.384 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.385+0000 7fa836ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.385+0000 7fa836ffd640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:43:53.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.385+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa820004eb0 con 0x7fa8400a98f0 2024-03-25T19:43:53.390 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.390+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82849159 0 0) 0x7fa84013b5c0 con 0x7fa8401430c0 2024-03-25T19:43:53.390 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.390+0000 7fa836ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.391 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.390+0000 7fa836ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.391 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.390+0000 7fa836ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.391 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.390+0000 7fa836ffd640 10 monclient(hunting): my global_id is 4295 2024-03-25T19:43:53.391 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.390+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8401d7df0 con 0x7fa8401430c0 2024-03-25T19:43:53.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.398+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533653240 0 0) 0x7fa8401d7f80 con 0x7fa84013f710 2024-03-25T19:43:53.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.398+0000 7fa836ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:53.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.398+0000 7fa836ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:53.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.398+0000 7fa836ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:53.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.398+0000 7fa836ffd640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:43:53.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.398+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa84013b5c0 con 0x7fa84013f710 2024-03-25T19:43:53.406 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.406+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 417823175 0 0) 0x7fa820004eb0 con 0x7fa8400a98f0 2024-03-25T19:43:53.406 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.406+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8401d7f80 con 0x7fa8400a98f0 2024-03-25T19:43:53.406 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.406+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa830001a30 con 0x7fa8400a98f0 2024-03-25T19:43:53.406 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.406+0000 7fa836ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.406 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.406+0000 7fa836ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:53.406 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.406+0000 7fa836ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:53.406 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:53.406 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:53.406 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:53.406 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:53.407 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.407 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:53.407 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:53.407 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:53.407 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:53.407 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:53.407 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.406+0000 7fa836ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:53.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.411+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 976536954 0 0) 0x7fa8401d7df0 con 0x7fa8401430c0 2024-03-25T19:43:53.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.411+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa820004eb0 con 0x7fa8401430c0 2024-03-25T19:43:53.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.411+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa838001e80 con 0x7fa8401430c0 2024-03-25T19:43:53.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.411+0000 7fa836ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.411 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.411+0000 7fa836ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:53.411 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.411+0000 7fa836ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:53.411 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:53.411 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:53.411 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:53.411 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:53.411 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.411 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:53.411 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:53.411 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:53.411 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:53.411 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:53.415 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.416+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3308500211 0 0) 0x7fa84013b5c0 con 0x7fa84013f710 2024-03-25T19:43:53.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.416+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8401d7df0 con 0x7fa84013f710 2024-03-25T19:43:53.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.416+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa828001830 con 0x7fa84013f710 2024-03-25T19:43:53.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.416+0000 7fa836ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.416+0000 7fa836ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:53.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.416+0000 7fa836ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:53.416 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:53.416 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:53.416 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:53.416 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:53.416 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.416 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:53.416 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:53.416 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:53.416 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:53.417 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:53.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.420+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1605195342 0 0) 0x7fa820004eb0 con 0x7fa8401430c0 2024-03-25T19:43:53.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.420+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 >> v1:172.21.15.98:6789/0 conn(0x7fa84013f710 legacy=0x7fa840133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.420+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 >> v1:172.21.15.81:6789/0 conn(0x7fa8400a98f0 legacy=0x7fa8401d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.420+0000 7fa836ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:53.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.420+0000 7fa836ffd640 1 monclient: found mon.c 2024-03-25T19:43:53.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.420+0000 7fa836ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:53.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.420+0000 7fa836ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:43:53.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.420+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84013b750 con 0x7fa8401430c0 2024-03-25T19:43:53.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.420+0000 7fa836ffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:53.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.420+0000 7fa836ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:53.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.420+0000 7fa836ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:53.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.420+0000 7fa844a25640 5 monclient: authenticate success, global_id 4295 2024-03-25T19:43:53.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.420+0000 7fa844a25640 10 monclient: _renew_subs 2024-03-25T19:43:53.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.420+0000 7fa844a25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:43:53.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.420+0000 7fa844a25640 1 -- 172.21.15.81:0/4064319290 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa840133710 con 0x7fa8401430c0 2024-03-25T19:43:53.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.421+0000 7fa844a25640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:53.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.421+0000 7fa844a25640 10 monclient: _renew_subs 2024-03-25T19:43:53.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.421+0000 7fa844a25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:43:53.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.421+0000 7fa844a25640 1 -- 172.21.15.81:0/4064319290 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8401357a0 con 0x7fa8401430c0 2024-03-25T19:43:53.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.421+0000 7fa844a25640 1 librados: init done 2024-03-25T19:43:53.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.422+0000 7fa844a25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:53.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.422+0000 7fa844a25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:53.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.422+0000 7fa844a25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:43:53.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.422+0000 7fa844a25640 1 -- 172.21.15.81:0/4064319290 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa840138fd0 con 0x7fa8401430c0 2024-03-25T19:43:53.728 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.728+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa838001fe0 con 0x7fa8401430c0 2024-03-25T19:43:53.728 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.728+0000 7fa836ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:53.728 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.728+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa838004e10 con 0x7fa8401430c0 2024-03-25T19:43:53.728 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.728+0000 7fa836ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:53.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.728+0000 7fa836ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:53.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.728+0000 7fa836ffd640 10 monclient: dump: 2024-03-25T19:43:53.729 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:43:53.729 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:43:53.729 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:53.729 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:43:53.729 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:53.729 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:43:53.729 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:43:53.729 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:43:53.729 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:43:53.729 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:43:53.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.728+0000 7fa83d59a640 4 set_mon_vals no callback set 2024-03-25T19:43:53.730 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.729+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fa83801ade0 con 0x7fa8401430c0 2024-03-25T19:43:53.730 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.730+0000 7fa83e59c640 10 monclient: get_auth_request con 0x7fa82006e0c0 auth_method 0 2024-03-25T19:43:53.730 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.730+0000 7fa836ffd640 10 client.4295.objecter ms_handle_connect 0x7fa82006e0c0 2024-03-25T19:43:53.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.736+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fa838054910 con 0x7fa8401430c0 2024-03-25T19:43:53.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.736+0000 7fa836ffd640 10 client.4295.objecter ms_dispatch 0x7fa840000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:53.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.736+0000 7fa836ffd640 3 client.4295.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:53.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.736+0000 7fa836ffd640 3 client.4295.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:53.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.736+0000 7fa836ffd640 20 client.4295.objecter dump_active .. 0 homeless 2024-03-25T19:43:53.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.736+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa838050420 con 0x7fa8401430c0 2024-03-25T19:43:53.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.736+0000 7fa836ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:53.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.736+0000 7fa836ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:53.978 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.978+0000 7fa844a25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:53.978 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.978+0000 7fa844a25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:53.978 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.978+0000 7fa844a25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:43:53.978 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:53.978+0000 7fa844a25640 1 -- 172.21.15.81:0/4064319290 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa840074cc0 con 0x7fa8401430c0 2024-03-25T19:43:54.032 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:54.033+0000 7fa836ffd640 1 -- 172.21.15.81:0/4064319290 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fa838052350 con 0x7fa8401430c0 2024-03-25T19:43:54.032 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:54.033+0000 7fa836ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:54.032 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:54.033+0000 7fa836ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:54.033 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:43:54.033 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:54.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:54.035+0000 7fa844a25640 10 librados: watch_flush enter 2024-03-25T19:43:54.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:54.035+0000 7fa844a25640 10 librados: watch_flush exit 2024-03-25T19:43:54.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:54.035+0000 7fa844a25640 20 client.4295.objecter shutdown clearing up homeless session... 2024-03-25T19:43:54.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:54.035+0000 7fa844a25640 10 client.4295.objecter successfully canceled tick 2024-03-25T19:43:54.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:54.035+0000 7fa844a25640 1 -- 172.21.15.81:0/4064319290 >> v1:172.21.15.98:6800/3980829775 conn(0x7fa82006e0c0 legacy=0x7fa820070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:54.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:54.035+0000 7fa844a25640 10 monclient: shutdown 2024-03-25T19:43:54.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:54.035+0000 7fa844a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:54.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:54.035+0000 7fa844a25640 1 -- 172.21.15.81:0/4064319290 >> v1:172.21.15.81:6790/0 conn(0x7fa8401430c0 legacy=0x7fa840135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:54.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:54.035+0000 7fa844a25640 1 -- 172.21.15.81:0/4064319290 shutdown_connections 2024-03-25T19:43:54.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:54.035+0000 7fa844a25640 1 -- 172.21.15.81:0/4064319290 >> 172.21.15.81:0/4064319290 conn(0x7fa8400b7ea0 msgr2=0x7fa840074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:54.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:54.035+0000 7fa844a25640 1 -- 172.21.15.81:0/4064319290 shutdown_connections 2024-03-25T19:43:54.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:54.035+0000 7fa844a25640 1 -- 172.21.15.81:0/4064319290 wait complete. 2024-03-25T19:43:54.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:43:54.035+0000 7fa844a25640 1 librados: shutdown 2024-03-25T19:43:54.047 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:56.894 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:19.529 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:56.895 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:02:19.536 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:56.897 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:19.531 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:56.905 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:02:19.547 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:56.911 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:19.545 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:56.940 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:02:19.582 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:56.950 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:02:19.591 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:56.959 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:02:19.601 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:56.998 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:02:19.640 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:00.853 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:00.968 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.968+0000 7f3ad4c2a640 10 monclient: get_monmap_and_config 2024-03-25T19:44:00.968 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.968+0000 7f3ad4c2a640 10 monclient: build_initial_monmap 2024-03-25T19:44:00.968 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.968+0000 7f3ad4c2a640 10 monclient: monmap: 2024-03-25T19:44:00.968 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:44:00.968 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:00.968 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:44:00.969524+0000 2024-03-25T19:44:00.968 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:44:00.969524+0000 2024-03-25T19:44:00.968 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:00.969 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:00.969 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:44:00.969 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:44:00.969 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:44:00.969 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:00.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.969+0000 7f3ad4c2a640 1 Processor -- start 2024-03-25T19:44:00.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.969+0000 7f3ad4c2a640 1 -- start start 2024-03-25T19:44:00.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.969+0000 7f3ad4c2a640 10 monclient: init 2024-03-25T19:44:00.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.969+0000 7f3ad4c2a640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:00.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.969+0000 7f3ad4c2a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:00.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.969+0000 7f3ad4c2a640 10 monclient(hunting): picked mon.noname-c con 0x7f3ad00a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:44:00.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.970+0000 7f3ad4c2a640 10 monclient(hunting): picked mon.noname-b con 0x7f3ad013f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:44:00.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.970+0000 7f3ad4c2a640 10 monclient(hunting): picked mon.noname-a con 0x7f3ad01430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:44:00.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.970+0000 7f3ad4c2a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad0142520 con 0x7f3ad01430c0 2024-03-25T19:44:00.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.970+0000 7f3ad4c2a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad00b28a0 con 0x7f3ad00a98f0 2024-03-25T19:44:00.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.970+0000 7f3ad4c2a640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad00b2a30 con 0x7f3ad013f710 2024-03-25T19:44:00.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.970+0000 7f3ad4c2a640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:00.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.970+0000 7f3ad4c2a640 10 monclient(hunting): authenticate will time out at 4076.051896s 2024-03-25T19:44:00.978 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.978+0000 7f3ace59c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3ad00a98f0 0x7f3ad00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:49996/0 (socket says 172.21.15.81:49996) 2024-03-25T19:44:00.978 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.978+0000 7f3ace59c640 1 -- 172.21.15.81:0/2836545209 learned_addr learned my addr 172.21.15.81:0/2836545209 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:44:00.988 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.989+0000 7f3acdd9b640 1 -- 172.21.15.81:0/2836545209 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037332803 0 0) 0x7f3ad0142520 con 0x7f3ad01430c0 2024-03-25T19:44:00.988 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.989+0000 7f3acdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:00.988 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.989+0000 7f3acdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:00.988 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.989+0000 7f3acdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:00.989 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.989+0000 7f3acdd9b640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:44:00.989 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.989+0000 7f3acdd9b640 1 -- 172.21.15.81:0/2836545209 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3abc003520 con 0x7f3ad01430c0 2024-03-25T19:44:00.999 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.999+0000 7f3acdd9b640 1 -- 172.21.15.81:0/2836545209 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490635949 0 0) 0x7f3ad00b28a0 con 0x7f3ad00a98f0 2024-03-25T19:44:00.999 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.999+0000 7f3acdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:44:00.999 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.999+0000 7f3acdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:00.999 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.999+0000 7f3acdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:00.999 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.999+0000 7f3acdd9b640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:44:00.999 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:00.999+0000 7f3acdd9b640 1 -- 172.21.15.81:0/2836545209 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ad0142520 con 0x7f3ad00a98f0 2024-03-25T19:44:01.008 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3acdd9b640 1 -- 172.21.15.81:0/2836545209 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1918816357 0 0) 0x7f3abc003520 con 0x7f3ad01430c0 2024-03-25T19:44:01.008 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3acdd9b640 1 -- 172.21.15.81:0/2836545209 >> v1:172.21.15.81:6790/0 conn(0x7f3ad013f710 legacy=0x7f3ad01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.008 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3acdd9b640 1 -- 172.21.15.81:0/2836545209 >> v1:172.21.15.98:6789/0 conn(0x7f3ad00a98f0 legacy=0x7f3ad00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.008 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3acdd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:44:01.008 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3acdd9b640 1 monclient: found mon.noname-a 2024-03-25T19:44:01.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3acdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:01.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3acdd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:44:01.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3acdd9b640 1 -- 172.21.15.81:0/2836545209 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad00b44c0 con 0x7f3ad01430c0 2024-03-25T19:44:01.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3acdd9b640 10 monclient: _finish_auth 0 2024-03-25T19:44:01.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3acdd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:44:01.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3acdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:01.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3acdd9b640 1 -- 172.21.15.81:0/2836545209 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3ac8001a10 con 0x7f3ad01430c0 2024-03-25T19:44:01.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3ad4c2a640 5 monclient: authenticate success, global_id 4314 2024-03-25T19:44:01.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3ad4c2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:01.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3acdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:01.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3acdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:01.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3acdd9b640 10 monclient: dump: 2024-03-25T19:44:01.010 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:01.010 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:01.010 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:01.010 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:01.010 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:01.010 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:01.010 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:01.010 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:01.010 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:01.010 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:01.010 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.008+0000 7f3ad4c2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:01.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.027+0000 7f3acdd9b640 1 -- 172.21.15.81:0/2836545209 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ac8001b70 con 0x7f3ad01430c0 2024-03-25T19:44:01.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.027+0000 7f3acdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:01.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.027+0000 7f3acdd9b640 1 -- 172.21.15.81:0/2836545209 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3ac8001f80 con 0x7f3ad01430c0 2024-03-25T19:44:01.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.027+0000 7f3acdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:01.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.027+0000 7f3acdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:01.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.027+0000 7f3acdd9b640 10 monclient: dump: 2024-03-25T19:44:01.028 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:01.028 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:01.028 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:01.028 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:01.028 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:01.028 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:01.028 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:01.028 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:01.028 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:01.028 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:01.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.028+0000 7f3ad4c2a640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:01.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.028+0000 7f3ad4c2a640 4 set_mon_vals no callback set 2024-03-25T19:44:01.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.028+0000 7f3ad4c2a640 10 monclient: shutdown 2024-03-25T19:44:01.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.028+0000 7f3ad4c2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:01.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.028+0000 7f3ad4c2a640 1 -- 172.21.15.81:0/2836545209 >> v1:172.21.15.81:6789/0 conn(0x7f3ad01430c0 legacy=0x7f3ad014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.029+0000 7f3ad4c2a640 1 -- 172.21.15.81:0/2836545209 shutdown_connections 2024-03-25T19:44:01.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.029+0000 7f3ad4c2a640 1 -- 172.21.15.81:0/2836545209 >> 172.21.15.81:0/2836545209 conn(0x7f3ad00b7ea0 msgr2=0x7f3ad00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:01.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.029+0000 7f3ad4c2a640 1 -- 172.21.15.81:0/2836545209 shutdown_connections 2024-03-25T19:44:01.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.029+0000 7f3ad4c2a640 1 -- 172.21.15.81:0/2836545209 wait complete. 2024-03-25T19:44:01.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.030+0000 7f3ad4c2a640 10 monclient: build_initial_monmap 2024-03-25T19:44:01.031 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.030+0000 7f3ad4c2a640 10 monclient: monmap: 2024-03-25T19:44:01.031 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:44:01.031 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:01.031 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:44:01.031 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:44:01.031 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:01.031 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:44:01.031 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:44:01.031 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:44:01.031 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:44:01.031 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:01.031 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.030+0000 7f3ad4c2a640 1 librados: starting msgr at 2024-03-25T19:44:01.031 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.030+0000 7f3ad4c2a640 1 librados: starting objecter 2024-03-25T19:44:01.032 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.030+0000 7f3ad4c2a640 1 Processor -- start 2024-03-25T19:44:01.032 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.030+0000 7f3ad4c2a640 1 -- start start 2024-03-25T19:44:01.032 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.030+0000 7f3ad4c2a640 1 librados: setting wanted keys 2024-03-25T19:44:01.032 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.030+0000 7f3ad4c2a640 1 librados: calling monclient init 2024-03-25T19:44:01.032 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.030+0000 7f3ad4c2a640 10 monclient: init 2024-03-25T19:44:01.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.031+0000 7f3ad4c2a640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:01.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.031+0000 7f3ad4c2a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:01.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.031+0000 7f3ad4c2a640 10 monclient(hunting): picked mon.noname-a con 0x7f3ad00a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:44:01.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.031+0000 7f3ad4c2a640 10 monclient(hunting): picked mon.noname-b con 0x7f3ad013f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:44:01.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.031+0000 7f3ad4c2a640 10 monclient(hunting): picked mon.noname-c con 0x7f3ad01430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:44:01.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.031+0000 7f3ad4c2a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad0139830 con 0x7f3ad00a98f0 2024-03-25T19:44:01.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.031+0000 7f3ad4c2a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad01db7d0 con 0x7f3ad013f710 2024-03-25T19:44:01.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.031+0000 7f3ad4c2a640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad01db410 con 0x7f3ad01430c0 2024-03-25T19:44:01.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.031+0000 7f3ad4c2a640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:01.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.031+0000 7f3ad4c2a640 10 monclient(hunting): authenticate will time out at 4076.113353s 2024-03-25T19:44:01.046 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.047+0000 7f3aced9d640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f3ad01430c0 0x7f3ad01d4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:57134/0 (socket says 172.21.15.81:57134) 2024-03-25T19:44:01.046 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.047+0000 7f3aced9d640 1 -- 172.21.15.81:0/645118915 learned_addr learned my addr 172.21.15.81:0/645118915 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:44:01.048 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.049+0000 7f3ac77fe640 10 client.?.objecter ms_handle_connect 0x7f3ad01430c0 2024-03-25T19:44:01.049 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.049+0000 7f3ac77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:01.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.051+0000 7f3ac77fe640 10 client.?.objecter ms_handle_connect 0x7f3ad00a98f0 2024-03-25T19:44:01.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.051+0000 7f3ac77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:01.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.060+0000 7f3ac77fe640 10 client.?.objecter ms_handle_connect 0x7f3ad013f710 2024-03-25T19:44:01.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.060+0000 7f3ac77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:01.062 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.062+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057949874 0 0) 0x7f3ad01db410 con 0x7f3ad01430c0 2024-03-25T19:44:01.062 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.062+0000 7f3ac77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:01.062 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.062+0000 7f3ac77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:01.062 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.062+0000 7f3ac77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:01.062 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.062+0000 7f3ac77fe640 10 monclient(hunting): my global_id is 4301 2024-03-25T19:44:01.062 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.062+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ab0004eb0 con 0x7f3ad01430c0 2024-03-25T19:44:01.067 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.068+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574503643 0 0) 0x7f3ad0139830 con 0x7f3ad00a98f0 2024-03-25T19:44:01.067 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.068+0000 7f3ac77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:01.067 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.068+0000 7f3ac77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:01.067 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.068+0000 7f3ac77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:01.067 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.068+0000 7f3ac77fe640 10 monclient(hunting): my global_id is 4317 2024-03-25T19:44:01.068 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.068+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ad01db410 con 0x7f3ad00a98f0 2024-03-25T19:44:01.073 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.073+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1728742168 0 0) 0x7f3ab0004eb0 con 0x7f3ad01430c0 2024-03-25T19:44:01.073 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.073+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ad0139830 con 0x7f3ad01430c0 2024-03-25T19:44:01.073 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.073+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3ac8001f80 con 0x7f3ad01430c0 2024-03-25T19:44:01.073 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.073+0000 7f3ac77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:01.073 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.073+0000 7f3ac77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:01.073 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.073+0000 7f3ac77fe640 10 monclient(hunting): dump: 2024-03-25T19:44:01.073 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:01.074 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:01.074 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:01.074 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:01.074 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:01.074 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:01.074 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:01.074 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:01.074 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:01.074 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:01.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.073+0000 7f3ac77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:01.079 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.080+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667152306 0 0) 0x7f3ad01db7d0 con 0x7f3ad013f710 2024-03-25T19:44:01.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.080+0000 7f3ac77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:01.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.080+0000 7f3ac77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:01.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.080+0000 7f3ac77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:01.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.080+0000 7f3ac77fe640 10 monclient(hunting): my global_id is 4315 2024-03-25T19:44:01.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.080+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ab0004eb0 con 0x7f3ad013f710 2024-03-25T19:44:01.088 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2653852181 0 0) 0x7f3ad0139830 con 0x7f3ad01430c0 2024-03-25T19:44:01.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 >> v1:172.21.15.98:6789/0 conn(0x7f3ad013f710 legacy=0x7f3ad00b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 >> v1:172.21.15.81:6789/0 conn(0x7f3ad00a98f0 legacy=0x7f3ad01dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ac77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:01.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ac77fe640 1 monclient: found mon.c 2024-03-25T19:44:01.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ac77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:01.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ac77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:44:01.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad01db5a0 con 0x7f3ad01430c0 2024-03-25T19:44:01.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ac77fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:01.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ac77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:01.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ac77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:01.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ad4c2a640 5 monclient: authenticate success, global_id 4301 2024-03-25T19:44:01.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ad4c2a640 10 monclient: _renew_subs 2024-03-25T19:44:01.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ad4c2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:44:01.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ad4c2a640 1 -- 172.21.15.81:0/645118915 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad00b1480 con 0x7f3ad01430c0 2024-03-25T19:44:01.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ad4c2a640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:01.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ad4c2a640 10 monclient: _renew_subs 2024-03-25T19:44:01.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ad4c2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:44:01.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ad4c2a640 1 -- 172.21.15.81:0/645118915 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad00b0b10 con 0x7f3ad01430c0 2024-03-25T19:44:01.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.088+0000 7f3ad4c2a640 1 librados: init done 2024-03-25T19:44:01.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.089+0000 7f3ad4c2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:01.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.089+0000 7f3ad4c2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:01.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.089+0000 7f3ad4c2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:44:01.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.089+0000 7f3ad4c2a640 1 -- 172.21.15.81:0/645118915 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad01db960 con 0x7f3ad01430c0 2024-03-25T19:44:01.396 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.396+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ac8003c20 con 0x7f3ad01430c0 2024-03-25T19:44:01.396 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.396+0000 7f3ac77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:01.396 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.396+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3ac8003f70 con 0x7f3ad01430c0 2024-03-25T19:44:01.396 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.396+0000 7f3ac77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:01.396 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.396+0000 7f3ac77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:01.396 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.396+0000 7f3ac77fe640 10 monclient: dump: 2024-03-25T19:44:01.396 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:01.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.396+0000 7f3acd59a640 4 set_mon_vals no callback set 2024-03-25T19:44:01.398 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.397+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f3ac8004190 con 0x7f3ad01430c0 2024-03-25T19:44:01.398 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.398+0000 7f3ace59c640 10 monclient: get_auth_request con 0x7f3ab006de00 auth_method 0 2024-03-25T19:44:01.398 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.398+0000 7f3ac77fe640 10 client.4301.objecter ms_handle_connect 0x7f3ab006de00 2024-03-25T19:44:01.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.402+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f3ac8054bb0 con 0x7f3ad01430c0 2024-03-25T19:44:01.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.402+0000 7f3ac77fe640 10 client.4301.objecter ms_dispatch 0x7f3ad0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:01.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.402+0000 7f3ac77fe640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:01.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.402+0000 7f3ac77fe640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:01.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.402+0000 7f3ac77fe640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-25T19:44:01.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.402+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3ac80506c0 con 0x7f3ad01430c0 2024-03-25T19:44:01.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.402+0000 7f3ac77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:01.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.402+0000 7f3ac77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:01.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.625+0000 7f3ad4c2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:01.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.625+0000 7f3ad4c2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:01.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.625+0000 7f3ad4c2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:44:01.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.625+0000 7f3ad4c2a640 1 -- 172.21.15.81:0/645118915 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ad00627b0 con 0x7f3ad01430c0 2024-03-25T19:44:01.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.647+0000 7f3ac77fe640 1 -- 172.21.15.81:0/645118915 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f3ac80525f0 con 0x7f3ad01430c0 2024-03-25T19:44:01.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.647+0000 7f3ac77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:01.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.647+0000 7f3ac77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:01.647 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:44:01.647 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:01.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.649+0000 7f3ad4c2a640 10 librados: watch_flush enter 2024-03-25T19:44:01.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.649+0000 7f3ad4c2a640 10 librados: watch_flush exit 2024-03-25T19:44:01.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.649+0000 7f3ad4c2a640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-25T19:44:01.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.649+0000 7f3ad4c2a640 10 client.4301.objecter successfully canceled tick 2024-03-25T19:44:01.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.649+0000 7f3ad4c2a640 1 -- 172.21.15.81:0/645118915 >> v1:172.21.15.98:6800/3980829775 conn(0x7f3ab006de00 legacy=0x7f3ab00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.649+0000 7f3ad4c2a640 10 monclient: shutdown 2024-03-25T19:44:01.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.649+0000 7f3ad4c2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:01.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.649+0000 7f3ad4c2a640 1 -- 172.21.15.81:0/645118915 >> v1:172.21.15.81:6790/0 conn(0x7f3ad01430c0 legacy=0x7f3ad01d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:01.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.649+0000 7f3ad4c2a640 1 -- 172.21.15.81:0/645118915 shutdown_connections 2024-03-25T19:44:01.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.649+0000 7f3ad4c2a640 1 -- 172.21.15.81:0/645118915 >> 172.21.15.81:0/645118915 conn(0x7f3ad00b7ea0 msgr2=0x7f3ad0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:01.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.649+0000 7f3ad4c2a640 1 -- 172.21.15.81:0/645118915 shutdown_connections 2024-03-25T19:44:01.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.649+0000 7f3ad4c2a640 1 -- 172.21.15.81:0/645118915 wait complete. 2024-03-25T19:44:01.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:01.649+0000 7f3ad4c2a640 1 librados: shutdown 2024-03-25T19:44:01.660 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:08.468 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:08.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.568+0000 7f8c28a8f640 10 monclient: get_monmap_and_config 2024-03-25T19:44:08.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.568+0000 7f8c28a8f640 10 monclient: build_initial_monmap 2024-03-25T19:44:08.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.568+0000 7f8c28a8f640 10 monclient: monmap: 2024-03-25T19:44:08.568 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:44:08.568 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:08.568 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:44:08.569409+0000 2024-03-25T19:44:08.568 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:44:08.569409+0000 2024-03-25T19:44:08.568 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:08.568 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:08.568 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:44:08.568 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:44:08.569 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:44:08.569 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:08.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.569+0000 7f8c28a8f640 1 Processor -- start 2024-03-25T19:44:08.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.569+0000 7f8c28a8f640 1 -- start start 2024-03-25T19:44:08.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.569+0000 7f8c28a8f640 10 monclient: init 2024-03-25T19:44:08.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.570+0000 7f8c28a8f640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:08.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.570+0000 7f8c28a8f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:08.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.570+0000 7f8c28a8f640 10 monclient(hunting): picked mon.noname-a con 0x7f8c240b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:44:08.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.570+0000 7f8c28a8f640 10 monclient(hunting): picked mon.noname-c con 0x7f8c240b1730 addr v1:172.21.15.98:6789/0 2024-03-25T19:44:08.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.570+0000 7f8c28a8f640 10 monclient(hunting): picked mon.noname-b con 0x7f8c2414be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:44:08.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.570+0000 7f8c28a8f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c240af940 con 0x7f8c240b4d60 2024-03-25T19:44:08.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.570+0000 7f8c28a8f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c240b0a70 con 0x7f8c240b1730 2024-03-25T19:44:08.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.570+0000 7f8c28a8f640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c240b0d50 con 0x7f8c2414be90 2024-03-25T19:44:08.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.570+0000 7f8c28a8f640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:08.571 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.570+0000 7f8c28a8f640 10 monclient(hunting): authenticate will time out at 4083.652429s 2024-03-25T19:44:08.579 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.580+0000 7f8c2259c640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f8c240b4d60 0x7f8c2413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:33018/0 (socket says 172.21.15.81:33018) 2024-03-25T19:44:08.579 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.580+0000 7f8c2259c640 1 -- 172.21.15.81:0/391183821 learned_addr learned my addr 172.21.15.81:0/391183821 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:44:08.594 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.594+0000 7f8c2159a640 1 -- 172.21.15.81:0/391183821 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717515713 0 0) 0x7f8c240af940 con 0x7f8c240b4d60 2024-03-25T19:44:08.594 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.594+0000 7f8c2159a640 10 monclient(hunting): _negotiate 2024-03-25T19:44:08.594 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.594+0000 7f8c2159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:08.594 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.594+0000 7f8c2159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:08.594 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.594+0000 7f8c2159a640 10 monclient(hunting): my global_id is 4320 2024-03-25T19:44:08.594 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.594+0000 7f8c2159a640 1 -- 172.21.15.81:0/391183821 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c0c003520 con 0x7f8c240b4d60 2024-03-25T19:44:08.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.596+0000 7f8c2159a640 1 -- 172.21.15.81:0/391183821 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480724372 0 0) 0x7f8c240b0d50 con 0x7f8c2414be90 2024-03-25T19:44:08.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.596+0000 7f8c2159a640 10 monclient(hunting): _negotiate 2024-03-25T19:44:08.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.596+0000 7f8c2159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:08.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.596+0000 7f8c2159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:08.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.596+0000 7f8c2159a640 10 monclient(hunting): my global_id is 4304 2024-03-25T19:44:08.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.596+0000 7f8c2159a640 1 -- 172.21.15.81:0/391183821 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c240af940 con 0x7f8c2414be90 2024-03-25T19:44:08.601 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.601+0000 7f8c2159a640 1 -- 172.21.15.81:0/391183821 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099221805 0 0) 0x7f8c240b0a70 con 0x7f8c240b1730 2024-03-25T19:44:08.601 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.601+0000 7f8c2159a640 10 monclient(hunting): _negotiate 2024-03-25T19:44:08.601 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.601+0000 7f8c2159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:08.601 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.601+0000 7f8c2159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:08.601 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.601+0000 7f8c2159a640 10 monclient(hunting): my global_id is 4318 2024-03-25T19:44:08.601 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.601+0000 7f8c2159a640 1 -- 172.21.15.81:0/391183821 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c240b0d50 con 0x7f8c240b1730 2024-03-25T19:44:08.607 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.607+0000 7f8c2159a640 1 -- 172.21.15.81:0/391183821 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1690340009 0 0) 0x7f8c0c003520 con 0x7f8c240b4d60 2024-03-25T19:44:08.607 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.607+0000 7f8c2159a640 1 -- 172.21.15.81:0/391183821 >> v1:172.21.15.81:6790/0 conn(0x7f8c2414be90 legacy=0x7f8c2414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:08.607 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.607+0000 7f8c2159a640 1 -- 172.21.15.81:0/391183821 >> v1:172.21.15.98:6789/0 conn(0x7f8c240b1730 legacy=0x7f8c240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.607+0000 7f8c2159a640 10 monclient: _finish_hunting 0 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.607+0000 7f8c2159a640 1 monclient: found mon.noname-a 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.607+0000 7f8c2159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.607+0000 7f8c2159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.607+0000 7f8c2159a640 1 -- 172.21.15.81:0/391183821 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c240b1030 con 0x7f8c240b4d60 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.607+0000 7f8c2159a640 10 monclient: _finish_auth 0 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.608+0000 7f8c2159a640 10 monclient: _check_auth_tickets 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.608+0000 7f8c2159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.608+0000 7f8c2159a640 1 -- 172.21.15.81:0/391183821 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8c14001690 con 0x7f8c240b4d60 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.608+0000 7f8c28a8f640 5 monclient: authenticate success, global_id 4320 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.608+0000 7f8c28a8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.608+0000 7f8c2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.608+0000 7f8c2159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:08.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.608+0000 7f8c2159a640 10 monclient: dump: 2024-03-25T19:44:08.609 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:08.609 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:08.609 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:08.609 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:08.609 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:08.609 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:08.609 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:08.609 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:08.609 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:08.609 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:08.609 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.608+0000 7f8c28a8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:08.620 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.620+0000 7f8c2159a640 1 -- 172.21.15.81:0/391183821 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c140017f0 con 0x7f8c240b4d60 2024-03-25T19:44:08.620 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.620+0000 7f8c2159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:08.620 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.620+0000 7f8c2159a640 1 -- 172.21.15.81:0/391183821 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8c14001c00 con 0x7f8c240b4d60 2024-03-25T19:44:08.620 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.620+0000 7f8c28a8f640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:08.620 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.620+0000 7f8c28a8f640 4 set_mon_vals no callback set 2024-03-25T19:44:08.620 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.620+0000 7f8c2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:08.620 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.620+0000 7f8c2159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:08.620 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.620+0000 7f8c2159a640 10 monclient: dump: 2024-03-25T19:44:08.620 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:08.620 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:08.620 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:08.620 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:08.621 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:08.621 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:08.621 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:08.621 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:08.621 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:08.621 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:08.623 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.621+0000 7f8c28a8f640 10 monclient: shutdown 2024-03-25T19:44:08.623 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.621+0000 7f8c28a8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:08.623 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.621+0000 7f8c28a8f640 1 -- 172.21.15.81:0/391183821 >> v1:172.21.15.81:6789/0 conn(0x7f8c240b4d60 legacy=0x7f8c2413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:08.623 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.621+0000 7f8c28a8f640 1 -- 172.21.15.81:0/391183821 shutdown_connections 2024-03-25T19:44:08.623 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.621+0000 7f8c28a8f640 1 -- 172.21.15.81:0/391183821 >> 172.21.15.81:0/391183821 conn(0x7f8c240617a0 msgr2=0x7f8c24061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:08.623 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.621+0000 7f8c28a8f640 1 -- 172.21.15.81:0/391183821 shutdown_connections 2024-03-25T19:44:08.623 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.621+0000 7f8c28a8f640 1 -- 172.21.15.81:0/391183821 wait complete. 2024-03-25T19:44:08.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.622+0000 7f8c28a8f640 10 monclient: build_initial_monmap 2024-03-25T19:44:08.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.622+0000 7f8c28a8f640 10 monclient: monmap: 2024-03-25T19:44:08.624 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:44:08.624 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:08.624 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:44:08.624 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:44:08.624 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:08.624 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:44:08.624 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:44:08.624 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:44:08.624 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:44:08.624 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:08.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.622+0000 7f8c28a8f640 1 librados: starting msgr at 2024-03-25T19:44:08.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.622+0000 7f8c28a8f640 1 librados: starting objecter 2024-03-25T19:44:08.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.623+0000 7f8c28a8f640 1 Processor -- start 2024-03-25T19:44:08.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.623+0000 7f8c28a8f640 1 -- start start 2024-03-25T19:44:08.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.623+0000 7f8c28a8f640 1 librados: setting wanted keys 2024-03-25T19:44:08.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.623+0000 7f8c28a8f640 1 librados: calling monclient init 2024-03-25T19:44:08.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.623+0000 7f8c28a8f640 10 monclient: init 2024-03-25T19:44:08.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.623+0000 7f8c28a8f640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:08.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.623+0000 7f8c28a8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:08.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.623+0000 7f8c28a8f640 10 monclient(hunting): picked mon.noname-b con 0x7f8c240b1730 addr v1:172.21.15.98:6789/0 2024-03-25T19:44:08.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.623+0000 7f8c28a8f640 10 monclient(hunting): picked mon.noname-c con 0x7f8c240b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:44:08.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.623+0000 7f8c28a8f640 10 monclient(hunting): picked mon.noname-a con 0x7f8c2414be90 addr v1:172.21.15.81:6789/0 2024-03-25T19:44:08.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.623+0000 7f8c28a8f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c24136540 con 0x7f8c2414be90 2024-03-25T19:44:08.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.623+0000 7f8c28a8f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c24138360 con 0x7f8c240b1730 2024-03-25T19:44:08.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.623+0000 7f8c28a8f640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c24137300 con 0x7f8c240b4d60 2024-03-25T19:44:08.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.623+0000 7f8c28a8f640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:08.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.623+0000 7f8c28a8f640 10 monclient(hunting): authenticate will time out at 4083.705653s 2024-03-25T19:44:08.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.627+0000 7f8c21d9b640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f8c240b4d60 0x7f8c241d7d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:38300/0 (socket says 172.21.15.81:38300) 2024-03-25T19:44:08.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.627+0000 7f8c21d9b640 1 -- 172.21.15.81:0/1128963719 learned_addr learned my addr 172.21.15.81:0/1128963719 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:44:08.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.630+0000 7f8c12ffd640 10 client.?.objecter ms_handle_connect 0x7f8c240b4d60 2024-03-25T19:44:08.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.630+0000 7f8c12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:08.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.636+0000 7f8c12ffd640 10 client.?.objecter ms_handle_connect 0x7f8c240b1730 2024-03-25T19:44:08.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.636+0000 7f8c12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:08.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.640+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093458942 0 0) 0x7f8c24138360 con 0x7f8c240b1730 2024-03-25T19:44:08.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.640+0000 7f8c12ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:08.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.640+0000 7f8c12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:08.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.640+0000 7f8c12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:08.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.640+0000 7f8c12ffd640 10 monclient(hunting): my global_id is 4321 2024-03-25T19:44:08.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.640+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c04004eb0 con 0x7f8c240b1730 2024-03-25T19:44:08.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.641+0000 7f8c12ffd640 10 client.?.objecter ms_handle_connect 0x7f8c2414be90 2024-03-25T19:44:08.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.641+0000 7f8c12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:08.652 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.652+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690704022 0 0) 0x7f8c24137300 con 0x7f8c240b4d60 2024-03-25T19:44:08.652 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.653+0000 7f8c12ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:08.652 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.653+0000 7f8c12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:08.652 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.653+0000 7f8c12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:08.652 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.653+0000 7f8c12ffd640 10 monclient(hunting): my global_id is 4307 2024-03-25T19:44:08.652 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.653+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c24138360 con 0x7f8c240b4d60 2024-03-25T19:44:08.655 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.655+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 348474996 0 0) 0x7f8c04004eb0 con 0x7f8c240b1730 2024-03-25T19:44:08.655 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.655+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c24137300 con 0x7f8c240b1730 2024-03-25T19:44:08.655 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.655+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8c14001b00 con 0x7f8c240b1730 2024-03-25T19:44:08.655 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.655+0000 7f8c12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:08.655 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.655+0000 7f8c12ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:08.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.655+0000 7f8c12ffd640 10 monclient(hunting): dump: 2024-03-25T19:44:08.656 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:08.656 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:08.656 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:08.656 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:08.656 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:08.656 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:08.656 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:08.656 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:08.656 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:08.656 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:08.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.655+0000 7f8c12ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:08.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.661+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323140054 0 0) 0x7f8c24136540 con 0x7f8c2414be90 2024-03-25T19:44:08.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.661+0000 7f8c12ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:08.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.661+0000 7f8c12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:08.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.661+0000 7f8c12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:08.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.661+0000 7f8c12ffd640 10 monclient(hunting): my global_id is 4323 2024-03-25T19:44:08.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.661+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c04004eb0 con 0x7f8c2414be90 2024-03-25T19:44:08.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.664+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 358116571 0 0) 0x7f8c24138360 con 0x7f8c240b4d60 2024-03-25T19:44:08.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.664+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8c24136540 con 0x7f8c240b4d60 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.669+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8c18001a30 con 0x7f8c240b4d60 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.669+0000 7f8c12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.669+0000 7f8c12ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.669+0000 7f8c12ffd640 10 monclient(hunting): dump: 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:08.669 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:08.670 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:08.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 49497283 0 0) 0x7f8c24137300 con 0x7f8c240b1730 2024-03-25T19:44:08.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 >> v1:172.21.15.81:6790/0 conn(0x7f8c240b4d60 legacy=0x7f8c241d7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:08.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 >> v1:172.21.15.81:6789/0 conn(0x7f8c2414be90 legacy=0x7f8c2412fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:08.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c12ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:08.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c12ffd640 1 monclient: found mon.b 2024-03-25T19:44:08.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:08.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c12ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:44:08.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c241d8450 con 0x7f8c240b1730 2024-03-25T19:44:08.676 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c12ffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:08.676 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c12ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:08.676 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:08.676 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c28a8f640 5 monclient: authenticate success, global_id 4321 2024-03-25T19:44:08.676 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c28a8f640 10 monclient: _renew_subs 2024-03-25T19:44:08.676 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c28a8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:44:08.676 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c28a8f640 1 -- 172.21.15.81:0/1128963719 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c24143900 con 0x7f8c240b1730 2024-03-25T19:44:08.676 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c28a8f640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:08.676 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c28a8f640 10 monclient: _renew_subs 2024-03-25T19:44:08.676 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c28a8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:44:08.676 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c28a8f640 1 -- 172.21.15.81:0/1128963719 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c241361c0 con 0x7f8c240b1730 2024-03-25T19:44:08.676 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.675+0000 7f8c28a8f640 1 librados: init done 2024-03-25T19:44:08.677 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.676+0000 7f8c28a8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:08.677 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.676+0000 7f8c28a8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:08.677 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.676+0000 7f8c28a8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:44:08.677 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.676+0000 7f8c28a8f640 1 -- 172.21.15.81:0/1128963719 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c241401f0 con 0x7f8c240b1730 2024-03-25T19:44:08.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.970+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8c14001220 con 0x7f8c240b1730 2024-03-25T19:44:08.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.970+0000 7f8c12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:08.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.970+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8c14004c30 con 0x7f8c240b1730 2024-03-25T19:44:08.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.970+0000 7f8c12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:08.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.970+0000 7f8c12ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:08.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.970+0000 7f8c12ffd640 10 monclient: dump: 2024-03-25T19:44:08.970 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:08.970 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:08.970 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:08.970 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:08.970 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:08.971 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:08.971 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:08.971 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:08.971 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:08.971 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:08.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.970+0000 7f8c20d99640 4 set_mon_vals no callback set 2024-03-25T19:44:08.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.971+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f8c1401ac00 con 0x7f8c240b1730 2024-03-25T19:44:08.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.972+0000 7f8c21d9b640 10 monclient: get_auth_request con 0x7f8c0406df40 auth_method 0 2024-03-25T19:44:08.972 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.972+0000 7f8c12ffd640 10 client.4321.objecter ms_handle_connect 0x7f8c0406df40 2024-03-25T19:44:08.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.977+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f8c14054840 con 0x7f8c240b1730 2024-03-25T19:44:08.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.977+0000 7f8c12ffd640 10 client.4321.objecter ms_dispatch 0x7f8c24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:08.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.977+0000 7f8c12ffd640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:08.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.977+0000 7f8c12ffd640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:08.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.977+0000 7f8c12ffd640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-25T19:44:08.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.977+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8c14050180 con 0x7f8c240b1730 2024-03-25T19:44:08.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.977+0000 7f8c12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:08.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:08.977+0000 7f8c12ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:09.219 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.219+0000 7f8c28a8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:09.219 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.219+0000 7f8c28a8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:09.219 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.219+0000 7f8c28a8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:44:09.219 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.219+0000 7f8c28a8f640 1 -- 172.21.15.81:0/1128963719 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c240627b0 con 0x7f8c240b1730 2024-03-25T19:44:09.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.249+0000 7f8c12ffd640 1 -- 172.21.15.81:0/1128963719 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f8c14052280 con 0x7f8c240b1730 2024-03-25T19:44:09.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.249+0000 7f8c12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:09.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.249+0000 7f8c12ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:09.249 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:44:09.249 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:09.251 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.251+0000 7f8c28a8f640 10 librados: watch_flush enter 2024-03-25T19:44:09.251 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.251+0000 7f8c28a8f640 10 librados: watch_flush exit 2024-03-25T19:44:09.251 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.251+0000 7f8c28a8f640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-25T19:44:09.251 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.251+0000 7f8c28a8f640 10 client.4321.objecter successfully canceled tick 2024-03-25T19:44:09.251 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.251+0000 7f8c28a8f640 1 -- 172.21.15.81:0/1128963719 >> v1:172.21.15.98:6800/3980829775 conn(0x7f8c0406df40 legacy=0x7f8c04070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:09.251 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.251+0000 7f8c28a8f640 10 monclient: shutdown 2024-03-25T19:44:09.251 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.251+0000 7f8c28a8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:09.251 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.251+0000 7f8c28a8f640 1 -- 172.21.15.81:0/1128963719 >> v1:172.21.15.98:6789/0 conn(0x7f8c240b1730 legacy=0x7f8c24136bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:09.251 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.251+0000 7f8c28a8f640 1 -- 172.21.15.81:0/1128963719 shutdown_connections 2024-03-25T19:44:09.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.251+0000 7f8c28a8f640 1 -- 172.21.15.81:0/1128963719 >> 172.21.15.81:0/1128963719 conn(0x7f8c240617a0 msgr2=0x7f8c240b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:09.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.251+0000 7f8c28a8f640 1 -- 172.21.15.81:0/1128963719 shutdown_connections 2024-03-25T19:44:09.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.251+0000 7f8c28a8f640 1 -- 172.21.15.81:0/1128963719 wait complete. 2024-03-25T19:44:09.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:09.251+0000 7f8c28a8f640 1 librados: shutdown 2024-03-25T19:44:09.263 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:16.069 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:16.183 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.184+0000 7fe8a0f54640 10 monclient: get_monmap_and_config 2024-03-25T19:44:16.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.184+0000 7fe8a0f54640 10 monclient: build_initial_monmap 2024-03-25T19:44:16.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.184+0000 7fe8a0f54640 10 monclient: monmap: 2024-03-25T19:44:16.184 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:44:16.184 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:16.184 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:44:16.185400+0000 2024-03-25T19:44:16.184 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:44:16.185400+0000 2024-03-25T19:44:16.184 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:16.184 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:16.184 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:44:16.184 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:44:16.184 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:44:16.184 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:16.186 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.185+0000 7fe8a0f54640 1 Processor -- start 2024-03-25T19:44:16.186 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.185+0000 7fe8a0f54640 1 -- start start 2024-03-25T19:44:16.186 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.185+0000 7fe8a0f54640 10 monclient: init 2024-03-25T19:44:16.186 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.186+0000 7fe8a0f54640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:16.186 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.186+0000 7fe8a0f54640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:16.186 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.186+0000 7fe8a0f54640 10 monclient(hunting): picked mon.noname-b con 0x7fe89c0a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:44:16.186 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.186+0000 7fe8a0f54640 10 monclient(hunting): picked mon.noname-a con 0x7fe89c13f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:44:16.186 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.186+0000 7fe8a0f54640 10 monclient(hunting): picked mon.noname-c con 0x7fe89c1430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:44:16.186 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.186+0000 7fe8a0f54640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c142520 con 0x7fe89c13f710 2024-03-25T19:44:16.186 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.186+0000 7fe8a0f54640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c0b28a0 con 0x7fe89c1430c0 2024-03-25T19:44:16.186 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.186+0000 7fe8a0f54640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c0b2a30 con 0x7fe89c0a98f0 2024-03-25T19:44:16.187 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.186+0000 7fe8a0f54640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:16.187 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.186+0000 7fe8a0f54640 10 monclient(hunting): authenticate will time out at 4091.268127s 2024-03-25T19:44:16.192 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.193+0000 7fe89a59c640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fe89c0a98f0 0x7fe89c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:43144/0 (socket says 172.21.15.81:43144) 2024-03-25T19:44:16.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.193+0000 7fe89a59c640 1 -- 172.21.15.81:0/3987497156 learned_addr learned my addr 172.21.15.81:0/3987497156 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:44:16.198 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.199+0000 7fe89959a640 1 -- 172.21.15.81:0/3987497156 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701328677 0 0) 0x7fe89c0b28a0 con 0x7fe89c1430c0 2024-03-25T19:44:16.198 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.199+0000 7fe89959a640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.198 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.199+0000 7fe89959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.198 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.199+0000 7fe89959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.199 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.199+0000 7fe89959a640 10 monclient(hunting): my global_id is 4324 2024-03-25T19:44:16.199 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.199+0000 7fe89959a640 1 -- 172.21.15.81:0/3987497156 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe884003520 con 0x7fe89c1430c0 2024-03-25T19:44:16.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.201+0000 7fe89959a640 1 -- 172.21.15.81:0/3987497156 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271435356 0 0) 0x7fe89c142520 con 0x7fe89c13f710 2024-03-25T19:44:16.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.201+0000 7fe89959a640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.201+0000 7fe89959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.201+0000 7fe89959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.201+0000 7fe89959a640 10 monclient(hunting): my global_id is 4326 2024-03-25T19:44:16.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.201+0000 7fe89959a640 1 -- 172.21.15.81:0/3987497156 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe89c0b28a0 con 0x7fe89c13f710 2024-03-25T19:44:16.207 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.208+0000 7fe89959a640 1 -- 172.21.15.81:0/3987497156 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2079027760 0 0) 0x7fe884003520 con 0x7fe89c1430c0 2024-03-25T19:44:16.207 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.208+0000 7fe89959a640 1 -- 172.21.15.81:0/3987497156 >> v1:172.21.15.81:6790/0 conn(0x7fe89c0a98f0 legacy=0x7fe89c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.208+0000 7fe89959a640 1 -- 172.21.15.81:0/3987497156 >> v1:172.21.15.81:6789/0 conn(0x7fe89c13f710 legacy=0x7fe89c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.208+0000 7fe89959a640 10 monclient: _finish_hunting 0 2024-03-25T19:44:16.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.208+0000 7fe89959a640 1 monclient: found mon.noname-c 2024-03-25T19:44:16.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.208+0000 7fe89959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:16.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.208+0000 7fe89959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:44:16.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.208+0000 7fe89959a640 1 -- 172.21.15.81:0/3987497156 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c0b44c0 con 0x7fe89c1430c0 2024-03-25T19:44:16.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.208+0000 7fe89959a640 10 monclient: _finish_auth 0 2024-03-25T19:44:16.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.208+0000 7fe89959a640 10 monclient: _check_auth_tickets 2024-03-25T19:44:16.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.208+0000 7fe89959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:16.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.208+0000 7fe89959a640 1 -- 172.21.15.81:0/3987497156 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe894001a10 con 0x7fe89c1430c0 2024-03-25T19:44:16.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.208+0000 7fe89959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:16.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.208+0000 7fe89959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:16.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.208+0000 7fe89959a640 10 monclient: dump: 2024-03-25T19:44:16.208 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:16.209 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:16.209 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:16.209 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:16.209 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:16.209 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:16.209 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:16.209 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:16.209 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:16.209 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:16.209 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.208+0000 7fe8a0f54640 5 monclient: authenticate success, global_id 4324 2024-03-25T19:44:16.209 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.208+0000 7fe8a0f54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:16.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.211+0000 7fe89959a640 1 -- 172.21.15.81:0/3987497156 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe894001b70 con 0x7fe89c1430c0 2024-03-25T19:44:16.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.211+0000 7fe89959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:16.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.211+0000 7fe8a0f54640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:16.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.211+0000 7fe8a0f54640 4 set_mon_vals no callback set 2024-03-25T19:44:16.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.212+0000 7fe8a0f54640 10 monclient: shutdown 2024-03-25T19:44:16.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.212+0000 7fe8a0f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:16.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.212+0000 7fe8a0f54640 1 -- 172.21.15.81:0/3987497156 >> v1:172.21.15.98:6789/0 conn(0x7fe89c1430c0 legacy=0x7fe89c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.212+0000 7fe8a0f54640 1 -- 172.21.15.81:0/3987497156 shutdown_connections 2024-03-25T19:44:16.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.212+0000 7fe8a0f54640 1 -- 172.21.15.81:0/3987497156 >> 172.21.15.81:0/3987497156 conn(0x7fe89c0b7ea0 msgr2=0x7fe89c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:16.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.212+0000 7fe8a0f54640 1 -- 172.21.15.81:0/3987497156 shutdown_connections 2024-03-25T19:44:16.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.212+0000 7fe8a0f54640 1 -- 172.21.15.81:0/3987497156 wait complete. 2024-03-25T19:44:16.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.213+0000 7fe8a0f54640 10 monclient: build_initial_monmap 2024-03-25T19:44:16.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.213+0000 7fe8a0f54640 10 monclient: monmap: 2024-03-25T19:44:16.213 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:44:16.213 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:16.213 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:44:16.213 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:44:16.213 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:16.213 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:44:16.213 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:44:16.213 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:44:16.213 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:44:16.213 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:16.214 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.214+0000 7fe8a0f54640 1 librados: starting msgr at 2024-03-25T19:44:16.214 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.214+0000 7fe8a0f54640 1 librados: starting objecter 2024-03-25T19:44:16.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.214+0000 7fe8a0f54640 1 Processor -- start 2024-03-25T19:44:16.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.214+0000 7fe8a0f54640 1 -- start start 2024-03-25T19:44:16.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.214+0000 7fe8a0f54640 1 librados: setting wanted keys 2024-03-25T19:44:16.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.214+0000 7fe8a0f54640 1 librados: calling monclient init 2024-03-25T19:44:16.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.214+0000 7fe8a0f54640 10 monclient: init 2024-03-25T19:44:16.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.214+0000 7fe8a0f54640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:16.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.214+0000 7fe8a0f54640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:16.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.214+0000 7fe8a0f54640 10 monclient(hunting): picked mon.noname-b con 0x7fe89c0a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:44:16.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.214+0000 7fe8a0f54640 10 monclient(hunting): picked mon.noname-a con 0x7fe89c13f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:44:16.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.215+0000 7fe8a0f54640 10 monclient(hunting): picked mon.noname-c con 0x7fe89c1430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:44:16.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.215+0000 7fe8a0f54640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c138ba0 con 0x7fe89c13f710 2024-03-25T19:44:16.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.215+0000 7fe8a0f54640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c0ab9e0 con 0x7fe89c0a98f0 2024-03-25T19:44:16.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.215+0000 7fe8a0f54640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c0b1590 con 0x7fe89c1430c0 2024-03-25T19:44:16.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.215+0000 7fe8a0f54640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:16.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.215+0000 7fe8a0f54640 10 monclient(hunting): authenticate will time out at 4091.296908s 2024-03-25T19:44:16.222 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.223+0000 7fe89ad9d640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fe89c1430c0 0x7fe89c1da8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:43196/0 (socket says 172.21.15.81:43196) 2024-03-25T19:44:16.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.223+0000 7fe89ad9d640 1 -- 172.21.15.81:0/3452425826 learned_addr learned my addr 172.21.15.81:0/3452425826 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:44:16.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.224+0000 7fe88affd640 10 client.?.objecter ms_handle_connect 0x7fe89c1430c0 2024-03-25T19:44:16.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.224+0000 7fe88affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:16.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.231+0000 7fe88affd640 1 -- 172.21.15.81:0/3452425826 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530329621 0 0) 0x7fe89c0b1590 con 0x7fe89c1430c0 2024-03-25T19:44:16.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.231+0000 7fe88affd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.231+0000 7fe88affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.231+0000 7fe88affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.231+0000 7fe88affd640 10 monclient(hunting): my global_id is 4313 2024-03-25T19:44:16.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.231+0000 7fe88affd640 1 -- 172.21.15.81:0/3452425826 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe87c004eb0 con 0x7fe89c1430c0 2024-03-25T19:44:16.235 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.235+0000 7fe88affd640 10 client.?.objecter ms_handle_connect 0x7fe89c13f710 2024-03-25T19:44:16.235 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.235+0000 7fe88affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:16.236 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.236+0000 7fe88affd640 10 client.?.objecter ms_handle_connect 0x7fe89c0a98f0 2024-03-25T19:44:16.236 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.236+0000 7fe88affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:16.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.239+0000 7fe88affd640 1 -- 172.21.15.81:0/3452425826 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3852051775 0 0) 0x7fe87c004eb0 con 0x7fe89c1430c0 2024-03-25T19:44:16.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.239+0000 7fe88affd640 1 -- 172.21.15.81:0/3452425826 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe89c0b1590 con 0x7fe89c1430c0 2024-03-25T19:44:16.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.239+0000 7fe88affd640 1 -- 172.21.15.81:0/3452425826 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe8940019d0 con 0x7fe89c1430c0 2024-03-25T19:44:16.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.239+0000 7fe88affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:16.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.239+0000 7fe88affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:16.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.239+0000 7fe88affd640 10 monclient(hunting): dump: 2024-03-25T19:44:16.239 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:16.239 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:16.239 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:16.239 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:16.239 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:16.239 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:16.239 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:16.239 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:16.239 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:16.239 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:16.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.239+0000 7fe88affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:16.244 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.244+0000 7fe88affd640 1 -- 172.21.15.81:0/3452425826 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648835614 0 0) 0x7fe89c138ba0 con 0x7fe89c13f710 2024-03-25T19:44:16.244 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.244+0000 7fe88affd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:16.244 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.244+0000 7fe88affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:16.244 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.244+0000 7fe88affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:16.244 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.244+0000 7fe88affd640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:44:16.244 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.244+0000 7fe88affd640 1 -- 172.21.15.81:0/3452425826 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe87c004eb0 con 0x7fe89c13f710 2024-03-25T19:44:16.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe88affd640 1 -- 172.21.15.81:0/3452425826 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3190698346 0 0) 0x7fe89c0b1590 con 0x7fe89c1430c0 2024-03-25T19:44:16.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe88affd640 1 -- 172.21.15.81:0/3452425826 >> v1:172.21.15.98:6789/0 conn(0x7fe89c0a98f0 legacy=0x7fe89c137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe88affd640 1 -- 172.21.15.81:0/3452425826 >> v1:172.21.15.81:6789/0 conn(0x7fe89c13f710 legacy=0x7fe89c133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe88affd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:16.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe88affd640 1 monclient: found mon.c 2024-03-25T19:44:16.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe88affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:16.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe88affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:44:16.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe88affd640 1 -- 172.21.15.81:0/3452425826 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c0b1720 con 0x7fe89c1430c0 2024-03-25T19:44:16.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe88affd640 10 monclient: _finish_auth 0 2024-03-25T19:44:16.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe88affd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:16.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe88affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:16.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe8a0f54640 5 monclient: authenticate success, global_id 4313 2024-03-25T19:44:16.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe8a0f54640 10 monclient: _renew_subs 2024-03-25T19:44:16.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe8a0f54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:44:16.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe8a0f54640 1 -- 172.21.15.81:0/3452425826 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89c134690 con 0x7fe89c1430c0 2024-03-25T19:44:16.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe8a0f54640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:16.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe8a0f54640 10 monclient: _renew_subs 2024-03-25T19:44:16.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe8a0f54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:44:16.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe8a0f54640 1 -- 172.21.15.81:0/3452425826 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89c1db000 con 0x7fe89c1430c0 2024-03-25T19:44:16.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.247+0000 7fe8a0f54640 1 librados: init done 2024-03-25T19:44:16.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.249+0000 7fe8a0f54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:16.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.249+0000 7fe8a0f54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:16.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.249+0000 7fe8a0f54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:44:16.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.249+0000 7fe8a0f54640 1 -- 172.21.15.81:0/3452425826 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89c138920 con 0x7fe89c1430c0 2024-03-25T19:44:16.563 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.563+0000 7fe88affd640 1 -- 172.21.15.81:0/3452425826 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe894003c20 con 0x7fe89c1430c0 2024-03-25T19:44:16.563 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.563+0000 7fe88affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:16.563 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.563+0000 7fe88affd640 1 -- 172.21.15.81:0/3452425826 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe894003ff0 con 0x7fe89c1430c0 2024-03-25T19:44:16.563 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.563+0000 7fe88affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:16.563 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.563+0000 7fe88affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:16.564 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.563+0000 7fe88affd640 10 monclient: dump: 2024-03-25T19:44:16.564 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:16.564 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:16.564 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:16.564 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:16.564 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:16.564 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:16.564 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:16.564 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:16.564 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:16.564 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:16.564 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.563+0000 7fe898d99640 4 set_mon_vals no callback set 2024-03-25T19:44:16.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.564+0000 7fe88affd640 1 -- 172.21.15.81:0/3452425826 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fe894004210 con 0x7fe89c1430c0 2024-03-25T19:44:16.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.565+0000 7fe89a59c640 10 monclient: get_auth_request con 0x7fe87c06dd90 auth_method 0 2024-03-25T19:44:16.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.565+0000 7fe88affd640 10 client.4313.objecter ms_handle_connect 0x7fe87c06dd90 2024-03-25T19:44:16.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.570+0000 7fe88affd640 1 -- 172.21.15.81:0/3452425826 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fe894054bd0 con 0x7fe89c1430c0 2024-03-25T19:44:16.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.570+0000 7fe88affd640 10 client.4313.objecter ms_dispatch 0x7fe89c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:16.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.570+0000 7fe88affd640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:16.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.570+0000 7fe88affd640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:16.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.570+0000 7fe88affd640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-25T19:44:16.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.570+0000 7fe88affd640 1 -- 172.21.15.81:0/3452425826 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8940506e0 con 0x7fe89c1430c0 2024-03-25T19:44:16.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.570+0000 7fe88affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:16.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.570+0000 7fe88affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:16.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.801+0000 7fe8a0f54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:16.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.801+0000 7fe8a0f54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:16.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.801+0000 7fe8a0f54640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:44:16.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.801+0000 7fe8a0f54640 1 -- 172.21.15.81:0/3452425826 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe89c0b18b0 con 0x7fe89c1430c0 2024-03-25T19:44:16.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.823+0000 7fe88affd640 1 -- 172.21.15.81:0/3452425826 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fe89c0b18b0 con 0x7fe89c1430c0 2024-03-25T19:44:16.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.824+0000 7fe88affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:16.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.824+0000 7fe88affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:16.823 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:44:16.824 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:16.826 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.826+0000 7fe8a0f54640 10 librados: watch_flush enter 2024-03-25T19:44:16.826 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.826+0000 7fe8a0f54640 10 librados: watch_flush exit 2024-03-25T19:44:16.826 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.826+0000 7fe8a0f54640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-25T19:44:16.826 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.826+0000 7fe8a0f54640 10 client.4313.objecter successfully canceled tick 2024-03-25T19:44:16.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.826+0000 7fe8a0f54640 1 -- 172.21.15.81:0/3452425826 >> v1:172.21.15.98:6800/3980829775 conn(0x7fe87c06dd90 legacy=0x7fe87c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.826+0000 7fe8a0f54640 10 monclient: shutdown 2024-03-25T19:44:16.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.826+0000 7fe8a0f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:16.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.826+0000 7fe8a0f54640 1 -- 172.21.15.81:0/3452425826 >> v1:172.21.15.81:6790/0 conn(0x7fe89c1430c0 legacy=0x7fe89c1da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.826+0000 7fe8a0f54640 1 -- 172.21.15.81:0/3452425826 shutdown_connections 2024-03-25T19:44:16.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.826+0000 7fe8a0f54640 1 -- 172.21.15.81:0/3452425826 >> 172.21.15.81:0/3452425826 conn(0x7fe89c0b7ea0 msgr2=0x7fe89c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:16.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.826+0000 7fe8a0f54640 1 -- 172.21.15.81:0/3452425826 shutdown_connections 2024-03-25T19:44:16.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.826+0000 7fe8a0f54640 1 -- 172.21.15.81:0/3452425826 wait complete. 2024-03-25T19:44:16.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:16.826+0000 7fe8a0f54640 1 librados: shutdown 2024-03-25T19:44:16.838 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:16.942 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:02:39.584 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:16.954 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:02:39.596 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:16.985 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:02:39.627 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:17.003 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:02:39.645 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:17.009 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:39.643 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:17.014 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:02:39.656 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:17.020 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:39.655 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:17.046 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:02:39.688 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:17.057 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:39.692 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:23.645 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:23.759 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.759+0000 7ff8a58f6640 10 monclient: get_monmap_and_config 2024-03-25T19:44:23.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.759+0000 7ff8a58f6640 10 monclient: build_initial_monmap 2024-03-25T19:44:23.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.759+0000 7ff8a58f6640 10 monclient: monmap: 2024-03-25T19:44:23.760 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:44:23.760 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:23.760 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:44:23.761338+0000 2024-03-25T19:44:23.760 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:44:23.761338+0000 2024-03-25T19:44:23.760 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:23.760 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:23.760 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:44:23.760 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:44:23.760 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:44:23.760 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:23.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.760+0000 7ff8a58f6640 1 Processor -- start 2024-03-25T19:44:23.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.760+0000 7ff8a58f6640 1 -- start start 2024-03-25T19:44:23.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.760+0000 7ff8a58f6640 10 monclient: init 2024-03-25T19:44:23.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.760+0000 7ff8a58f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:23.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.760+0000 7ff8a58f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:23.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.760+0000 7ff8a58f6640 10 monclient(hunting): picked mon.noname-c con 0x7ff8a00b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:44:23.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.760+0000 7ff8a58f6640 10 monclient(hunting): picked mon.noname-a con 0x7ff8a00b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:44:23.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.760+0000 7ff8a58f6640 10 monclient(hunting): picked mon.noname-b con 0x7ff8a014be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:44:23.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.760+0000 7ff8a58f6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a00af940 con 0x7ff8a00b1730 2024-03-25T19:44:23.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.761+0000 7ff8a58f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a00b0a70 con 0x7ff8a00b4d60 2024-03-25T19:44:23.763 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.761+0000 7ff8a58f6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a00b0d50 con 0x7ff8a014be90 2024-03-25T19:44:23.763 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.761+0000 7ff8a58f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:23.763 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.761+0000 7ff8a58f6640 10 monclient(hunting): authenticate will time out at 4098.843895s 2024-03-25T19:44:23.779 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.779+0000 7ff89f7fe640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7ff8a014be90 0x7ff8a014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:44690/0 (socket says 172.21.15.81:44690) 2024-03-25T19:44:23.779 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.779+0000 7ff89f7fe640 1 -- 172.21.15.81:0/4159612178 learned_addr learned my addr 172.21.15.81:0/4159612178 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:44:23.795 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.794+0000 7ff89dffb640 1 -- 172.21.15.81:0/4159612178 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818942662 0 0) 0x7ff8a00b0d50 con 0x7ff8a014be90 2024-03-25T19:44:23.795 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.794+0000 7ff89dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:23.795 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.794+0000 7ff89dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:23.795 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.794+0000 7ff89dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:23.795 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.794+0000 7ff89dffb640 10 monclient(hunting): my global_id is 4316 2024-03-25T19:44:23.795 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.794+0000 7ff89dffb640 1 -- 172.21.15.81:0/4159612178 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff888003520 con 0x7ff8a014be90 2024-03-25T19:44:23.799 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.798+0000 7ff89dffb640 1 -- 172.21.15.81:0/4159612178 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384969123 0 0) 0x7ff8a00b0a70 con 0x7ff8a00b4d60 2024-03-25T19:44:23.799 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.798+0000 7ff89dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:23.799 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.798+0000 7ff89dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:23.799 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.798+0000 7ff89dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:23.799 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.798+0000 7ff89dffb640 10 monclient(hunting): my global_id is 4330 2024-03-25T19:44:23.799 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.798+0000 7ff89dffb640 1 -- 172.21.15.81:0/4159612178 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a00b0d50 con 0x7ff8a00b4d60 2024-03-25T19:44:23.811 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.811+0000 7ff89dffb640 1 -- 172.21.15.81:0/4159612178 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 544273306 0 0) 0x7ff888003520 con 0x7ff8a014be90 2024-03-25T19:44:23.812 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.811+0000 7ff89dffb640 1 -- 172.21.15.81:0/4159612178 >> v1:172.21.15.98:6789/0 conn(0x7ff8a00b4d60 legacy=0x7ff8a013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:23.812 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.811+0000 7ff89dffb640 1 -- 172.21.15.81:0/4159612178 >> v1:172.21.15.81:6789/0 conn(0x7ff8a00b1730 legacy=0x7ff8a00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:23.812 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.811+0000 7ff89dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:44:23.812 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.811+0000 7ff89dffb640 1 monclient: found mon.noname-b 2024-03-25T19:44:23.812 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.811+0000 7ff89dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:23.812 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.811+0000 7ff89dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:44:23.812 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.811+0000 7ff89dffb640 1 -- 172.21.15.81:0/4159612178 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a00b1030 con 0x7ff8a014be90 2024-03-25T19:44:23.812 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.811+0000 7ff89dffb640 10 monclient: _finish_auth 0 2024-03-25T19:44:23.812 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.811+0000 7ff89dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:44:23.812 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.811+0000 7ff89dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:23.812 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.811+0000 7ff89dffb640 1 -- 172.21.15.81:0/4159612178 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7ff898001a10 con 0x7ff8a014be90 2024-03-25T19:44:23.812 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.811+0000 7ff89dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:23.812 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.811+0000 7ff89dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:23.813 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.811+0000 7ff89dffb640 10 monclient: dump: 2024-03-25T19:44:23.813 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:23.813 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:23.813 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:23.813 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:23.813 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:23.813 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:23.813 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:23.813 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:23.813 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:23.813 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:23.813 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.811+0000 7ff8a58f6640 5 monclient: authenticate success, global_id 4316 2024-03-25T19:44:23.813 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.811+0000 7ff8a58f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:23.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.826+0000 7ff89dffb640 1 -- 172.21.15.81:0/4159612178 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff898001b70 con 0x7ff8a014be90 2024-03-25T19:44:23.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.826+0000 7ff89dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:23.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.826+0000 7ff89dffb640 1 -- 172.21.15.81:0/4159612178 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7ff898001f80 con 0x7ff8a014be90 2024-03-25T19:44:23.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.826+0000 7ff8a58f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:23.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.826+0000 7ff8a58f6640 4 set_mon_vals no callback set 2024-03-25T19:44:23.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.826+0000 7ff89dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:23.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.826+0000 7ff89dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:23.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.826+0000 7ff89dffb640 10 monclient: dump: 2024-03-25T19:44:23.827 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:23.827 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:23.827 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:23.827 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:23.827 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:23.828 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:23.828 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:23.828 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:23.828 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:23.828 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:23.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.827+0000 7ff8a58f6640 10 monclient: shutdown 2024-03-25T19:44:23.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.827+0000 7ff8a58f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:23.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.827+0000 7ff8a58f6640 1 -- 172.21.15.81:0/4159612178 >> v1:172.21.15.81:6790/0 conn(0x7ff8a014be90 legacy=0x7ff8a014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:23.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.827+0000 7ff8a58f6640 1 -- 172.21.15.81:0/4159612178 shutdown_connections 2024-03-25T19:44:23.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.827+0000 7ff8a58f6640 1 -- 172.21.15.81:0/4159612178 >> 172.21.15.81:0/4159612178 conn(0x7ff8a00617a0 msgr2=0x7ff8a0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:23.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.827+0000 7ff8a58f6640 1 -- 172.21.15.81:0/4159612178 shutdown_connections 2024-03-25T19:44:23.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.827+0000 7ff8a58f6640 1 -- 172.21.15.81:0/4159612178 wait complete. 2024-03-25T19:44:23.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.828+0000 7ff8a58f6640 10 monclient: build_initial_monmap 2024-03-25T19:44:23.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.828+0000 7ff8a58f6640 10 monclient: monmap: 2024-03-25T19:44:23.831 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:44:23.831 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:23.831 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:44:23.831 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:44:23.831 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:23.831 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:44:23.831 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:44:23.831 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:44:23.831 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:44:23.831 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:23.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.828+0000 7ff8a58f6640 1 librados: starting msgr at 2024-03-25T19:44:23.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.828+0000 7ff8a58f6640 1 librados: starting objecter 2024-03-25T19:44:23.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.828+0000 7ff8a58f6640 1 Processor -- start 2024-03-25T19:44:23.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.828+0000 7ff8a58f6640 1 -- start start 2024-03-25T19:44:23.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.828+0000 7ff8a58f6640 1 librados: setting wanted keys 2024-03-25T19:44:23.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.828+0000 7ff8a58f6640 1 librados: calling monclient init 2024-03-25T19:44:23.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.828+0000 7ff8a58f6640 10 monclient: init 2024-03-25T19:44:23.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.829+0000 7ff8a58f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:23.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.829+0000 7ff8a58f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:23.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.829+0000 7ff8a58f6640 10 monclient(hunting): picked mon.noname-c con 0x7ff8a00b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:44:23.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.829+0000 7ff8a58f6640 10 monclient(hunting): picked mon.noname-a con 0x7ff8a00b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:44:23.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.829+0000 7ff8a58f6640 10 monclient(hunting): picked mon.noname-b con 0x7ff8a014be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:44:23.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.829+0000 7ff8a58f6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a01db040 con 0x7ff8a00b4d60 2024-03-25T19:44:23.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.829+0000 7ff8a58f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a01daca0 con 0x7ff8a014be90 2024-03-25T19:44:23.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.829+0000 7ff8a58f6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a01364e0 con 0x7ff8a00b1730 2024-03-25T19:44:23.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.829+0000 7ff8a58f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:23.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.829+0000 7ff8a58f6640 10 monclient(hunting): authenticate will time out at 4098.912329s 2024-03-25T19:44:23.835 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.834+0000 7ff89e7fc640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ff8a00b4d60 0x7ff8a0137830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:56344/0 (socket says 172.21.15.81:56344) 2024-03-25T19:44:23.835 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.834+0000 7ff89e7fc640 1 -- 172.21.15.81:0/1497350022 learned_addr learned my addr 172.21.15.81:0/1497350022 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:44:23.835 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.835+0000 7ff887fff640 10 client.?.objecter ms_handle_connect 0x7ff8a00b4d60 2024-03-25T19:44:23.835 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.835+0000 7ff887fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:23.842 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.841+0000 7ff887fff640 10 client.?.objecter ms_handle_connect 0x7ff8a014be90 2024-03-25T19:44:23.842 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.841+0000 7ff887fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:23.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.851+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389452129 0 0) 0x7ff8a01daca0 con 0x7ff8a014be90 2024-03-25T19:44:23.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.851+0000 7ff887fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:23.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.851+0000 7ff887fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:23.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.851+0000 7ff887fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:23.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.851+0000 7ff887fff640 10 monclient(hunting): my global_id is 4333 2024-03-25T19:44:23.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.851+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff88c004eb0 con 0x7ff8a014be90 2024-03-25T19:44:23.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.851+0000 7ff887fff640 10 client.?.objecter ms_handle_connect 0x7ff8a00b1730 2024-03-25T19:44:23.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.851+0000 7ff887fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:23.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.852+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199850821 0 0) 0x7ff8a01db040 con 0x7ff8a00b4d60 2024-03-25T19:44:23.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.852+0000 7ff887fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:23.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.852+0000 7ff887fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:23.853 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.852+0000 7ff887fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:23.853 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.852+0000 7ff887fff640 10 monclient(hunting): my global_id is 4335 2024-03-25T19:44:23.853 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.852+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a01daca0 con 0x7ff8a00b4d60 2024-03-25T19:44:23.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.862+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446754892 0 0) 0x7ff8a01364e0 con 0x7ff8a00b1730 2024-03-25T19:44:23.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.862+0000 7ff887fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:23.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.862+0000 7ff887fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:23.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.862+0000 7ff887fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:23.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.862+0000 7ff887fff640 10 monclient(hunting): my global_id is 4319 2024-03-25T19:44:23.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.862+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a01db040 con 0x7ff8a00b1730 2024-03-25T19:44:23.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.866+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4276969477 0 0) 0x7ff8a01daca0 con 0x7ff8a00b4d60 2024-03-25T19:44:23.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.866+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8a01364e0 con 0x7ff8a00b4d60 2024-03-25T19:44:23.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.866+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7ff894001970 con 0x7ff8a00b4d60 2024-03-25T19:44:23.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.866+0000 7ff887fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:23.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.866+0000 7ff887fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:23.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.866+0000 7ff887fff640 10 monclient(hunting): dump: 2024-03-25T19:44:23.867 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:23.867 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:23.867 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:23.867 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:23.867 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:23.867 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:23.867 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:23.867 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:23.867 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:23.867 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:23.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.866+0000 7ff887fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:23.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.871+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2620683020 0 0) 0x7ff88c004eb0 con 0x7ff8a014be90 2024-03-25T19:44:23.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.871+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8a01daca0 con 0x7ff8a014be90 2024-03-25T19:44:23.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.871+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7ff898001ec0 con 0x7ff8a014be90 2024-03-25T19:44:23.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.871+0000 7ff887fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:23.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.871+0000 7ff887fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:23.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.871+0000 7ff887fff640 10 monclient(hunting): dump: 2024-03-25T19:44:23.872 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:23.872 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:23.872 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:23.872 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:23.872 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:23.872 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:23.873 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:23.873 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:23.873 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:23.873 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:23.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2681547417 0 0) 0x7ff8a01364e0 con 0x7ff8a00b4d60 2024-03-25T19:44:23.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 >> v1:172.21.15.81:6790/0 conn(0x7ff8a00b1730 legacy=0x7ff8a01db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:23.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 >> v1:172.21.15.98:6789/0 conn(0x7ff8a014be90 legacy=0x7ff8a0137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:23.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff887fff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:23.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff887fff640 1 monclient: found mon.a 2024-03-25T19:44:23.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff887fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:23.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff887fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:44:23.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a0136670 con 0x7ff8a00b4d60 2024-03-25T19:44:23.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff887fff640 10 monclient: _finish_auth 0 2024-03-25T19:44:23.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff887fff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:23.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff887fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:23.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff8a58f6640 5 monclient: authenticate success, global_id 4335 2024-03-25T19:44:23.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff8a58f6640 10 monclient: _renew_subs 2024-03-25T19:44:23.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff8a58f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:44:23.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff8a58f6640 1 -- 172.21.15.81:0/1497350022 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a0142850 con 0x7ff8a00b4d60 2024-03-25T19:44:23.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff8a58f6640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:23.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff8a58f6640 10 monclient: _renew_subs 2024-03-25T19:44:23.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff8a58f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:44:23.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff8a58f6640 1 -- 172.21.15.81:0/1497350022 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a0135f50 con 0x7ff8a00b4d60 2024-03-25T19:44:23.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.872+0000 7ff8a58f6640 1 librados: init done 2024-03-25T19:44:23.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.874+0000 7ff8a58f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:23.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.874+0000 7ff8a58f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:23.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.874+0000 7ff8a58f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:44:23.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:23.874+0000 7ff8a58f6640 1 -- 172.21.15.81:0/1497350022 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a01db1d0 con 0x7ff8a00b4d60 2024-03-25T19:44:24.165 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.165+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff894003610 con 0x7ff8a00b4d60 2024-03-25T19:44:24.166 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.165+0000 7ff887fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:24.166 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.165+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7ff8940039a0 con 0x7ff8a00b4d60 2024-03-25T19:44:24.166 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.165+0000 7ff887fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:24.166 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.165+0000 7ff887fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:24.166 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.165+0000 7ff887fff640 10 monclient: dump: 2024-03-25T19:44:24.166 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:24.166 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:24.166 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:24.166 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:24.166 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:24.166 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:24.166 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:24.166 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:24.167 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:24.167 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:24.167 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.165+0000 7ff89d7fa640 4 set_mon_vals no callback set 2024-03-25T19:44:24.167 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.165+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7ff894003bc0 con 0x7ff8a00b4d60 2024-03-25T19:44:24.167 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.166+0000 7ff89effd640 10 monclient: get_auth_request con 0x7ff88c06df70 auth_method 0 2024-03-25T19:44:24.167 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.166+0000 7ff887fff640 10 client.4335.objecter ms_handle_connect 0x7ff88c06df70 2024-03-25T19:44:24.172 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.171+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7ff894054790 con 0x7ff8a00b4d60 2024-03-25T19:44:24.172 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.171+0000 7ff887fff640 10 client.4335.objecter ms_dispatch 0x7ff8a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:24.172 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.171+0000 7ff887fff640 3 client.4335.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:24.172 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.171+0000 7ff887fff640 3 client.4335.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:24.172 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.171+0000 7ff887fff640 20 client.4335.objecter dump_active .. 0 homeless 2024-03-25T19:44:24.172 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.171+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8940502a0 con 0x7ff8a00b4d60 2024-03-25T19:44:24.172 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.171+0000 7ff887fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:24.172 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.171+0000 7ff887fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:24.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.432+0000 7ff8a58f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:24.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.433+0000 7ff8a58f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:24.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.433+0000 7ff8a58f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:44:24.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.433+0000 7ff8a58f6640 1 -- 172.21.15.81:0/1497350022 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8a01dae30 con 0x7ff8a00b4d60 2024-03-25T19:44:24.460 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.459+0000 7ff887fff640 1 -- 172.21.15.81:0/1497350022 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7ff8940521d0 con 0x7ff8a00b4d60 2024-03-25T19:44:24.460 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.459+0000 7ff887fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:24.460 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.459+0000 7ff887fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:24.460 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:44:24.460 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:24.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.461+0000 7ff8a58f6640 10 librados: watch_flush enter 2024-03-25T19:44:24.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.462+0000 7ff8a58f6640 10 librados: watch_flush exit 2024-03-25T19:44:24.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.462+0000 7ff8a58f6640 20 client.4335.objecter shutdown clearing up homeless session... 2024-03-25T19:44:24.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.462+0000 7ff8a58f6640 10 client.4335.objecter successfully canceled tick 2024-03-25T19:44:24.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.462+0000 7ff8a58f6640 1 -- 172.21.15.81:0/1497350022 >> v1:172.21.15.98:6800/3980829775 conn(0x7ff88c06df70 legacy=0x7ff88c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.462+0000 7ff8a58f6640 10 monclient: shutdown 2024-03-25T19:44:24.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.462+0000 7ff8a58f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:24.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.462+0000 7ff8a58f6640 1 -- 172.21.15.81:0/1497350022 >> v1:172.21.15.81:6789/0 conn(0x7ff8a00b4d60 legacy=0x7ff8a0137830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:24.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.462+0000 7ff8a58f6640 1 -- 172.21.15.81:0/1497350022 shutdown_connections 2024-03-25T19:44:24.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.462+0000 7ff8a58f6640 1 -- 172.21.15.81:0/1497350022 >> 172.21.15.81:0/1497350022 conn(0x7ff8a00617a0 msgr2=0x7ff8a00b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:24.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.462+0000 7ff8a58f6640 1 -- 172.21.15.81:0/1497350022 shutdown_connections 2024-03-25T19:44:24.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.462+0000 7ff8a58f6640 1 -- 172.21.15.81:0/1497350022 wait complete. 2024-03-25T19:44:24.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:24.462+0000 7ff8a58f6640 1 librados: shutdown 2024-03-25T19:44:24.475 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:31.281 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:31.398 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.398+0000 7f8274406640 10 monclient: get_monmap_and_config 2024-03-25T19:44:31.398 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.398+0000 7f8274406640 10 monclient: build_initial_monmap 2024-03-25T19:44:31.398 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.398+0000 7f8274406640 10 monclient: monmap: 2024-03-25T19:44:31.398 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:44:31.399 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:31.399 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:44:31.399802+0000 2024-03-25T19:44:31.399 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:44:31.399802+0000 2024-03-25T19:44:31.399 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:31.399 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:31.399 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:44:31.399 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:44:31.399 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:44:31.399 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:31.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.399+0000 7f8274406640 1 Processor -- start 2024-03-25T19:44:31.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.399+0000 7f8274406640 1 -- start start 2024-03-25T19:44:31.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.399+0000 7f8274406640 10 monclient: init 2024-03-25T19:44:31.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.400+0000 7f8274406640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:31.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.400+0000 7f8274406640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:31.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.400+0000 7f8274406640 10 monclient(hunting): picked mon.noname-a con 0x7f826c0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:44:31.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.400+0000 7f8274406640 10 monclient(hunting): picked mon.noname-c con 0x7f826c13f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:44:31.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.400+0000 7f8274406640 10 monclient(hunting): picked mon.noname-b con 0x7f826c1430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:44:31.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.400+0000 7f8274406640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c142520 con 0x7f826c0a98f0 2024-03-25T19:44:31.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.400+0000 7f8274406640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c0b28a0 con 0x7f826c13f710 2024-03-25T19:44:31.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.400+0000 7f8274406640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c0b2a30 con 0x7f826c1430c0 2024-03-25T19:44:31.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.400+0000 7f8274406640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:31.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.400+0000 7f8274406640 10 monclient(hunting): authenticate will time out at 4106.482751s 2024-03-25T19:44:31.409 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.410+0000 7f82719a1640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f826c13f710 0x7f826c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:42966/0 (socket says 172.21.15.81:42966) 2024-03-25T19:44:31.409 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.410+0000 7f82719a1640 1 -- 172.21.15.81:0/1520864872 learned_addr learned my addr 172.21.15.81:0/1520864872 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:44:31.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.420+0000 7f82711a0640 1 -- 172.21.15.81:0/1520864872 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1964378705 0 0) 0x7f826c0b28a0 con 0x7f826c13f710 2024-03-25T19:44:31.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.420+0000 7f82711a0640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.420+0000 7f82711a0640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.420+0000 7f82711a0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.420+0000 7f82711a0640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:44:31.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.420+0000 7f82711a0640 1 -- 172.21.15.81:0/1520864872 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8258003520 con 0x7f826c13f710 2024-03-25T19:44:31.424 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.425+0000 7f82711a0640 1 -- 172.21.15.81:0/1520864872 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553063962 0 0) 0x7f826c0b2a30 con 0x7f826c1430c0 2024-03-25T19:44:31.424 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.425+0000 7f82711a0640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.424 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.425+0000 7f82711a0640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.424 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.425+0000 7f82711a0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.425+0000 7f82711a0640 10 monclient(hunting): my global_id is 4322 2024-03-25T19:44:31.425 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.425+0000 7f82711a0640 1 -- 172.21.15.81:0/1520864872 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f826c0b28a0 con 0x7f826c1430c0 2024-03-25T19:44:31.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.429+0000 7f82711a0640 1 -- 172.21.15.81:0/1520864872 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599933334 0 0) 0x7f826c142520 con 0x7f826c0a98f0 2024-03-25T19:44:31.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.429+0000 7f82711a0640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.429+0000 7f82711a0640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.429+0000 7f82711a0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.429+0000 7f82711a0640 10 monclient(hunting): my global_id is 4338 2024-03-25T19:44:31.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.429+0000 7f82711a0640 1 -- 172.21.15.81:0/1520864872 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f826c0b2a30 con 0x7f826c0a98f0 2024-03-25T19:44:31.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.433+0000 7f82711a0640 1 -- 172.21.15.81:0/1520864872 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 462774188 0 0) 0x7f8258003520 con 0x7f826c13f710 2024-03-25T19:44:31.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.433+0000 7f82711a0640 1 -- 172.21.15.81:0/1520864872 >> v1:172.21.15.81:6790/0 conn(0x7f826c1430c0 legacy=0x7f826c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.433+0000 7f82711a0640 1 -- 172.21.15.81:0/1520864872 >> v1:172.21.15.81:6789/0 conn(0x7f826c0a98f0 legacy=0x7f826c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.433+0000 7f82711a0640 10 monclient: _finish_hunting 0 2024-03-25T19:44:31.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.433+0000 7f82711a0640 1 monclient: found mon.noname-c 2024-03-25T19:44:31.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.433+0000 7f82711a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:31.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.433+0000 7f82711a0640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:44:31.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.433+0000 7f82711a0640 1 -- 172.21.15.81:0/1520864872 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c0b44c0 con 0x7f826c13f710 2024-03-25T19:44:31.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.433+0000 7f82711a0640 10 monclient: _finish_auth 0 2024-03-25T19:44:31.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.433+0000 7f82711a0640 10 monclient: _check_auth_tickets 2024-03-25T19:44:31.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.433+0000 7f82711a0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:31.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.433+0000 7f82711a0640 1 -- 172.21.15.81:0/1520864872 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f825c001690 con 0x7f826c13f710 2024-03-25T19:44:31.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.433+0000 7f82711a0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:31.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.433+0000 7f82711a0640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:31.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.433+0000 7f82711a0640 10 monclient: dump: 2024-03-25T19:44:31.434 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:31.434 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:31.434 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:31.434 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:31.434 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:31.435 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:31.435 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:31.435 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:31.435 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:31.435 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:31.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.433+0000 7f8274406640 5 monclient: authenticate success, global_id 4336 2024-03-25T19:44:31.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.433+0000 7f8274406640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:31.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.436+0000 7f82711a0640 1 -- 172.21.15.81:0/1520864872 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f825c0017f0 con 0x7f826c13f710 2024-03-25T19:44:31.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.436+0000 7f82711a0640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:31.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.436+0000 7f82711a0640 1 -- 172.21.15.81:0/1520864872 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f825c001c00 con 0x7f826c13f710 2024-03-25T19:44:31.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.436+0000 7f82711a0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:31.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.436+0000 7f82711a0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:31.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.436+0000 7f82711a0640 10 monclient: dump: 2024-03-25T19:44:31.436 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:31.436 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:31.436 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:31.436 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:31.437 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:31.437 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:31.437 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:31.437 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:31.437 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:31.437 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:31.437 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.436+0000 7f8274406640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:31.437 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.436+0000 7f8274406640 4 set_mon_vals no callback set 2024-03-25T19:44:31.438 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.437+0000 7f8274406640 10 monclient: shutdown 2024-03-25T19:44:31.438 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.437+0000 7f8274406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:31.438 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.437+0000 7f8274406640 1 -- 172.21.15.81:0/1520864872 >> v1:172.21.15.98:6789/0 conn(0x7f826c13f710 legacy=0x7f826c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.438 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.437+0000 7f8274406640 1 -- 172.21.15.81:0/1520864872 shutdown_connections 2024-03-25T19:44:31.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.437+0000 7f8274406640 1 -- 172.21.15.81:0/1520864872 >> 172.21.15.81:0/1520864872 conn(0x7f826c0b7ea0 msgr2=0x7f826c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:31.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.437+0000 7f8274406640 1 -- 172.21.15.81:0/1520864872 shutdown_connections 2024-03-25T19:44:31.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.438+0000 7f8274406640 1 -- 172.21.15.81:0/1520864872 wait complete. 2024-03-25T19:44:31.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.438+0000 7f8274406640 10 monclient: build_initial_monmap 2024-03-25T19:44:31.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.438+0000 7f8274406640 10 monclient: monmap: 2024-03-25T19:44:31.439 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:44:31.439 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:31.439 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:44:31.439 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:44:31.439 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:31.439 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:44:31.439 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:44:31.439 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:44:31.439 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:44:31.439 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:31.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.439+0000 7f8274406640 1 librados: starting msgr at 2024-03-25T19:44:31.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.439+0000 7f8274406640 1 librados: starting objecter 2024-03-25T19:44:31.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.439+0000 7f8274406640 1 Processor -- start 2024-03-25T19:44:31.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.439+0000 7f8274406640 1 -- start start 2024-03-25T19:44:31.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.439+0000 7f8274406640 1 librados: setting wanted keys 2024-03-25T19:44:31.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.439+0000 7f8274406640 1 librados: calling monclient init 2024-03-25T19:44:31.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.439+0000 7f8274406640 10 monclient: init 2024-03-25T19:44:31.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.439+0000 7f8274406640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:31.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.439+0000 7f8274406640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:31.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.439+0000 7f8274406640 10 monclient(hunting): picked mon.noname-c con 0x7f826c0a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:44:31.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.439+0000 7f8274406640 10 monclient(hunting): picked mon.noname-a con 0x7f826c13f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:44:31.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.440+0000 7f8274406640 10 monclient(hunting): picked mon.noname-b con 0x7f826c1430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:44:31.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.440+0000 7f8274406640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c0abdc0 con 0x7f826c13f710 2024-03-25T19:44:31.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.440+0000 7f8274406640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c1d7730 con 0x7f826c1430c0 2024-03-25T19:44:31.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.440+0000 7f8274406640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c138f10 con 0x7f826c0a98f0 2024-03-25T19:44:31.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.440+0000 7f8274406640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:31.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.440+0000 7f8274406640 10 monclient(hunting): authenticate will time out at 4106.521870s 2024-03-25T19:44:31.447 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.448+0000 7f82729a3640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f826c1430c0 0x7f826c1d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:43004/0 (socket says 172.21.15.81:43004) 2024-03-25T19:44:31.447 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.448+0000 7f82729a3640 1 -- 172.21.15.81:0/2158178076 learned_addr learned my addr 172.21.15.81:0/2158178076 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:44:31.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.450+0000 7f8262ffd640 10 client.?.objecter ms_handle_connect 0x7f826c1430c0 2024-03-25T19:44:31.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.450+0000 7f8262ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:31.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.457+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168318832 0 0) 0x7f826c1d7730 con 0x7f826c1430c0 2024-03-25T19:44:31.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.457+0000 7f8262ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.457+0000 7f8262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.457+0000 7f8262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.457+0000 7f8262ffd640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:44:31.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.457+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8250004eb0 con 0x7f826c1430c0 2024-03-25T19:44:31.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.465+0000 7f8262ffd640 10 client.?.objecter ms_handle_connect 0x7f826c0a98f0 2024-03-25T19:44:31.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.465+0000 7f8262ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:31.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.469+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 848083987 0 0) 0x7f8250004eb0 con 0x7f826c1430c0 2024-03-25T19:44:31.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.469+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f826c1d7730 con 0x7f826c1430c0 2024-03-25T19:44:31.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.469+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8264001c90 con 0x7f826c1430c0 2024-03-25T19:44:31.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.469+0000 7f8262ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:31.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.469+0000 7f8262ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:31.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.469+0000 7f8262ffd640 10 monclient(hunting): dump: 2024-03-25T19:44:31.469 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:31.469 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:31.470 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:31.470 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:31.470 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:31.470 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:31.470 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:31.470 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:31.470 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:31.470 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:31.470 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.469+0000 7f8262ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:31.472 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.473+0000 7f8262ffd640 10 client.?.objecter ms_handle_connect 0x7f826c13f710 2024-03-25T19:44:31.472 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.473+0000 7f8262ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:31.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.486+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287137435 0 0) 0x7f826c138f10 con 0x7f826c0a98f0 2024-03-25T19:44:31.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.486+0000 7f8262ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.486+0000 7f8262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.486+0000 7f8262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.486+0000 7f8262ffd640 10 monclient(hunting): my global_id is 4325 2024-03-25T19:44:31.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.486+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8250004eb0 con 0x7f826c0a98f0 2024-03-25T19:44:31.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.490+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000827976 0 0) 0x7f826c0abdc0 con 0x7f826c13f710 2024-03-25T19:44:31.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.490+0000 7f8262ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.490+0000 7f8262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.490+0000 7f8262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.490+0000 7f8262ffd640 10 monclient(hunting): my global_id is 4341 2024-03-25T19:44:31.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.490+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f826c138f10 con 0x7f826c13f710 2024-03-25T19:44:31.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.570+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 536872332 0 0) 0x7f826c138f10 con 0x7f826c13f710 2024-03-25T19:44:31.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.570+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f826c0abdc0 con 0x7f826c13f710 2024-03-25T19:44:31.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.570+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f825c001b00 con 0x7f826c13f710 2024-03-25T19:44:31.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.570+0000 7f8262ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:31.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.570+0000 7f8262ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:31.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.570+0000 7f8262ffd640 10 monclient(hunting): dump: 2024-03-25T19:44:31.570 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:31.570 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:31.570 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:31.570 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:31.570 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:31.570 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:31.570 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:31.571 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:31.571 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:31.571 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:31.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3366352482 0 0) 0x7f826c0abdc0 con 0x7f826c13f710 2024-03-25T19:44:31.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 >> v1:172.21.15.81:6790/0 conn(0x7f826c0a98f0 legacy=0x7f826c138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 >> v1:172.21.15.98:6789/0 conn(0x7f826c1430c0 legacy=0x7f826c1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8262ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:31.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8262ffd640 1 monclient: found mon.a 2024-03-25T19:44:31.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8262ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:31.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8262ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:44:31.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c1390a0 con 0x7f826c13f710 2024-03-25T19:44:31.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8262ffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:31.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8262ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:31.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8262ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:31.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8274406640 5 monclient: authenticate success, global_id 4341 2024-03-25T19:44:31.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8274406640 10 monclient: _renew_subs 2024-03-25T19:44:31.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8274406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:44:31.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8274406640 1 -- 172.21.15.81:0/2158178076 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826c1d8320 con 0x7f826c13f710 2024-03-25T19:44:31.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8274406640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:31.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8274406640 10 monclient: _renew_subs 2024-03-25T19:44:31.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8274406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:44:31.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8274406640 1 -- 172.21.15.81:0/2158178076 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f826c1d43c0 con 0x7f826c13f710 2024-03-25T19:44:31.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.605+0000 7f8274406640 1 librados: init done 2024-03-25T19:44:31.607 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.606+0000 7f8274406640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:31.607 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.606+0000 7f8274406640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:31.607 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.606+0000 7f8274406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:44:31.607 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.606+0000 7f8274406640 1 -- 172.21.15.81:0/2158178076 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f826c0627b0 con 0x7f826c13f710 2024-03-25T19:44:31.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.965+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f825c001240 con 0x7f826c13f710 2024-03-25T19:44:31.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.965+0000 7f8262ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:31.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.965+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f825c004a90 con 0x7f826c13f710 2024-03-25T19:44:31.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.965+0000 7f8262ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:31.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.965+0000 7f8262ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:31.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.965+0000 7f8262ffd640 10 monclient: dump: 2024-03-25T19:44:31.965 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:31.965 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:31.965 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:31.965 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:31.966 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:31.966 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:31.966 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:31.966 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:31.966 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:31.966 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:31.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.965+0000 7f827099f640 4 set_mon_vals no callback set 2024-03-25T19:44:31.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.966+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f825c01aa60 con 0x7f826c13f710 2024-03-25T19:44:31.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.967+0000 7f82721a2640 10 monclient: get_auth_request con 0x7f825006df30 auth_method 0 2024-03-25T19:44:31.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.967+0000 7f8262ffd640 10 client.4341.objecter ms_handle_connect 0x7f825006df30 2024-03-25T19:44:31.972 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.972+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f825c054790 con 0x7f826c13f710 2024-03-25T19:44:31.972 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.972+0000 7f8262ffd640 10 client.4341.objecter ms_dispatch 0x7f826c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:31.972 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.972+0000 7f8262ffd640 3 client.4341.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:31.972 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.972+0000 7f8262ffd640 3 client.4341.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:31.972 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.973+0000 7f8262ffd640 20 client.4341.objecter dump_active .. 0 homeless 2024-03-25T19:44:31.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.973+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f825c0500d0 con 0x7f826c13f710 2024-03-25T19:44:31.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.973+0000 7f8262ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:31.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:31.973+0000 7f8262ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:32.224 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.224+0000 7f8274406640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:32.224 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.224+0000 7f8274406640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:32.224 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.224+0000 7f8274406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:44:32.224 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.224+0000 7f8274406640 1 -- 172.21.15.81:0/2158178076 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f826c13fdc0 con 0x7f826c13f710 2024-03-25T19:44:32.283 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.284+0000 7f8262ffd640 1 -- 172.21.15.81:0/2158178076 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f825c0521d0 con 0x7f826c13f710 2024-03-25T19:44:32.284 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.284+0000 7f8262ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:32.284 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.284+0000 7f8262ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:32.284 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:44:32.284 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:32.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.286+0000 7f8274406640 10 librados: watch_flush enter 2024-03-25T19:44:32.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.286+0000 7f8274406640 10 librados: watch_flush exit 2024-03-25T19:44:32.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.286+0000 7f8274406640 20 client.4341.objecter shutdown clearing up homeless session... 2024-03-25T19:44:32.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.286+0000 7f8274406640 10 client.4341.objecter successfully canceled tick 2024-03-25T19:44:32.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.286+0000 7f8274406640 1 -- 172.21.15.81:0/2158178076 >> v1:172.21.15.98:6800/3980829775 conn(0x7f825006df30 legacy=0x7f82500703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:32.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.286+0000 7f8274406640 10 monclient: shutdown 2024-03-25T19:44:32.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.286+0000 7f8274406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:32.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.286+0000 7f8274406640 1 -- 172.21.15.81:0/2158178076 >> v1:172.21.15.81:6789/0 conn(0x7f826c13f710 legacy=0x7f826c1d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:32.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.286+0000 7f8274406640 1 -- 172.21.15.81:0/2158178076 shutdown_connections 2024-03-25T19:44:32.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.286+0000 7f8274406640 1 -- 172.21.15.81:0/2158178076 >> 172.21.15.81:0/2158178076 conn(0x7f826c0b7ea0 msgr2=0x7f826c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:32.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.287+0000 7f8274406640 1 -- 172.21.15.81:0/2158178076 shutdown_connections 2024-03-25T19:44:32.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.287+0000 7f8274406640 1 -- 172.21.15.81:0/2158178076 wait complete. 2024-03-25T19:44:32.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:32.287+0000 7f8274406640 1 librados: shutdown 2024-03-25T19:44:32.298 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:36.960 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:02:59.602 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:37.011 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:02:59.653 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:37.030 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:02:59.672 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:37.056 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:59.691 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:37.066 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:59.700 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:37.068 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:02:59.710 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:37.076 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:02:59.718 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:37.103 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:02:59.737 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:37.110 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:02:59.752 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:39.104 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:39.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.221+0000 7f104eb8d640 10 monclient: get_monmap_and_config 2024-03-25T19:44:39.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.221+0000 7f104eb8d640 10 monclient: build_initial_monmap 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.221+0000 7f104eb8d640 10 monclient: monmap: 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:44:39.222186+0000 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:44:39.222186+0000 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:44:39.221 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:39.222 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.222+0000 7f104eb8d640 1 Processor -- start 2024-03-25T19:44:39.222 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.222+0000 7f104eb8d640 1 -- start start 2024-03-25T19:44:39.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.222+0000 7f104eb8d640 10 monclient: init 2024-03-25T19:44:39.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.222+0000 7f104eb8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:39.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.222+0000 7f104eb8d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:39.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.222+0000 7f104eb8d640 10 monclient(hunting): picked mon.noname-c con 0x7f10480b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:44:39.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.223+0000 7f104eb8d640 10 monclient(hunting): picked mon.noname-b con 0x7f10480b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:44:39.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.223+0000 7f104eb8d640 10 monclient(hunting): picked mon.noname-a con 0x7f104814be90 addr v1:172.21.15.81:6789/0 2024-03-25T19:44:39.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.223+0000 7f104eb8d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10480af940 con 0x7f104814be90 2024-03-25T19:44:39.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.223+0000 7f104eb8d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10480b0a70 con 0x7f10480b4d60 2024-03-25T19:44:39.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.223+0000 7f104eb8d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10480b0d50 con 0x7f10480b1730 2024-03-25T19:44:39.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.223+0000 7f104eb8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:39.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.223+0000 7f104eb8d640 10 monclient(hunting): authenticate will time out at 4114.304909s 2024-03-25T19:44:39.235 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.236+0000 7f1047fff640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f10480b1730 0x7f10480b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:35456/0 (socket says 172.21.15.81:35456) 2024-03-25T19:44:39.236 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.236+0000 7f104d12a640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f104814be90 0x7f104814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:39506/0 (socket says 172.21.15.81:39506) 2024-03-25T19:44:39.236 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.236+0000 7f1047fff640 1 -- 172.21.15.81:0/1587859520 learned_addr learned my addr 172.21.15.81:0/1587859520 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:44:39.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.247+0000 7f10477fe640 1 -- 172.21.15.81:0/1587859520 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851584137 0 0) 0x7f10480b0d50 con 0x7f10480b1730 2024-03-25T19:44:39.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.247+0000 7f10477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.247+0000 7f10477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.247+0000 7f10477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.247+0000 7f10477fe640 10 monclient(hunting): my global_id is 4328 2024-03-25T19:44:39.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.247+0000 7f10477fe640 1 -- 172.21.15.81:0/1587859520 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1038003520 con 0x7f10480b1730 2024-03-25T19:44:39.251 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.252+0000 7f10477fe640 1 -- 172.21.15.81:0/1587859520 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310969019 0 0) 0x7f10480af940 con 0x7f104814be90 2024-03-25T19:44:39.251 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.252+0000 7f10477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.252+0000 7f10477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.252+0000 7f10477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.252+0000 7f10477fe640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:44:39.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.252+0000 7f10477fe640 1 -- 172.21.15.81:0/1587859520 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1038003840 con 0x7f104814be90 2024-03-25T19:44:39.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.254+0000 7f10477fe640 1 -- 172.21.15.81:0/1587859520 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992113037 0 0) 0x7f10480b0a70 con 0x7f10480b4d60 2024-03-25T19:44:39.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.254+0000 7f10477fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.254+0000 7f10477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.254 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.254+0000 7f10477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.254 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.254+0000 7f10477fe640 10 monclient(hunting): my global_id is 4342 2024-03-25T19:44:39.254 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.254+0000 7f10477fe640 1 -- 172.21.15.81:0/1587859520 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10480af940 con 0x7f10480b4d60 2024-03-25T19:44:39.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.261+0000 7f10477fe640 1 -- 172.21.15.81:0/1587859520 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3869763054 0 0) 0x7f1038003520 con 0x7f10480b1730 2024-03-25T19:44:39.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.261+0000 7f10477fe640 1 -- 172.21.15.81:0/1587859520 >> v1:172.21.15.98:6789/0 conn(0x7f10480b4d60 legacy=0x7f104813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.261+0000 7f10477fe640 1 -- 172.21.15.81:0/1587859520 >> v1:172.21.15.81:6789/0 conn(0x7f104814be90 legacy=0x7f104814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.261+0000 7f10477fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:39.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.261+0000 7f10477fe640 1 monclient: found mon.noname-b 2024-03-25T19:44:39.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.261+0000 7f10477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:39.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.261+0000 7f10477fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:44:39.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.261+0000 7f10477fe640 1 -- 172.21.15.81:0/1587859520 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10480b1030 con 0x7f10480b1730 2024-03-25T19:44:39.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.261+0000 7f10477fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:39.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.261+0000 7f10477fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.261+0000 7f10477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.261+0000 7f10477fe640 1 -- 172.21.15.81:0/1587859520 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1034001670 con 0x7f10480b1730 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.261+0000 7f10477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.261+0000 7f10477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.261+0000 7f10477fe640 10 monclient: dump: 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:39.262 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:39.263 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:39.263 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:39.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.261+0000 7f104eb8d640 5 monclient: authenticate success, global_id 4328 2024-03-25T19:44:39.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.261+0000 7f104eb8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:39.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.265+0000 7f10477fe640 1 -- 172.21.15.81:0/1587859520 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10340017d0 con 0x7f10480b1730 2024-03-25T19:44:39.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.265+0000 7f10477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:39.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.265+0000 7f10477fe640 1 -- 172.21.15.81:0/1587859520 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1034001be0 con 0x7f10480b1730 2024-03-25T19:44:39.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.265+0000 7f104eb8d640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:39.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.265+0000 7f104eb8d640 4 set_mon_vals no callback set 2024-03-25T19:44:39.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.265+0000 7f10477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:39.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.265+0000 7f10477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:39.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.265+0000 7f10477fe640 10 monclient: dump: 2024-03-25T19:44:39.266 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:39.266 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:39.266 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:39.266 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:39.266 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:39.266 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:39.266 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:39.266 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:39.266 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:39.266 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:39.268 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.266+0000 7f104eb8d640 10 monclient: shutdown 2024-03-25T19:44:39.268 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.266+0000 7f104eb8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:39.268 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.266+0000 7f104eb8d640 1 -- 172.21.15.81:0/1587859520 >> v1:172.21.15.81:6790/0 conn(0x7f10480b1730 legacy=0x7f10480b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.268 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.266+0000 7f104eb8d640 1 -- 172.21.15.81:0/1587859520 shutdown_connections 2024-03-25T19:44:39.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.266+0000 7f104eb8d640 1 -- 172.21.15.81:0/1587859520 >> 172.21.15.81:0/1587859520 conn(0x7f10480617a0 msgr2=0x7f1048061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:39.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.266+0000 7f104eb8d640 1 -- 172.21.15.81:0/1587859520 shutdown_connections 2024-03-25T19:44:39.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.266+0000 7f104eb8d640 1 -- 172.21.15.81:0/1587859520 wait complete. 2024-03-25T19:44:39.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.267+0000 7f104eb8d640 10 monclient: build_initial_monmap 2024-03-25T19:44:39.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.267+0000 7f104eb8d640 10 monclient: monmap: 2024-03-25T19:44:39.269 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:44:39.269 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:39.269 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:44:39.269 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:44:39.269 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:39.269 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:44:39.269 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:44:39.269 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:44:39.270 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:44:39.270 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:39.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.267+0000 7f104eb8d640 1 librados: starting msgr at 2024-03-25T19:44:39.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.267+0000 7f104eb8d640 1 librados: starting objecter 2024-03-25T19:44:39.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.268+0000 7f104eb8d640 1 Processor -- start 2024-03-25T19:44:39.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.268+0000 7f104eb8d640 1 -- start start 2024-03-25T19:44:39.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.268+0000 7f104eb8d640 1 librados: setting wanted keys 2024-03-25T19:44:39.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.268+0000 7f104eb8d640 1 librados: calling monclient init 2024-03-25T19:44:39.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.268+0000 7f104eb8d640 10 monclient: init 2024-03-25T19:44:39.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.268+0000 7f104eb8d640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:39.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.268+0000 7f104eb8d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:39.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.268+0000 7f104eb8d640 10 monclient(hunting): picked mon.noname-a con 0x7f10480b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:44:39.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.268+0000 7f104eb8d640 10 monclient(hunting): picked mon.noname-c con 0x7f10480b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:44:39.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.268+0000 7f104eb8d640 10 monclient(hunting): picked mon.noname-b con 0x7f104814be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:44:39.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.268+0000 7f104eb8d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1048136250 con 0x7f10480b1730 2024-03-25T19:44:39.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.268+0000 7f104eb8d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1048136af0 con 0x7f104814be90 2024-03-25T19:44:39.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.268+0000 7f104eb8d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1048136c80 con 0x7f10480b4d60 2024-03-25T19:44:39.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.268+0000 7f104eb8d640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:39.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.268+0000 7f104eb8d640 10 monclient(hunting): authenticate will time out at 4114.350777s 2024-03-25T19:44:39.285 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.285+0000 7f104d12a640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f104814be90 0x7f10481e2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:42204/0 (socket says 172.21.15.81:42204) 2024-03-25T19:44:39.285 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.285+0000 7f104d12a640 1 -- 172.21.15.81:0/1929282703 learned_addr learned my addr 172.21.15.81:0/1929282703 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:44:39.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.288+0000 7f10457fa640 10 client.?.objecter ms_handle_connect 0x7f104814be90 2024-03-25T19:44:39.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.288+0000 7f10457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:39.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.289+0000 7f10457fa640 10 client.?.objecter ms_handle_connect 0x7f10480b4d60 2024-03-25T19:44:39.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.289+0000 7f10457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:39.292 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.293+0000 7f10457fa640 10 client.?.objecter ms_handle_connect 0x7f10480b1730 2024-03-25T19:44:39.292 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.293+0000 7f10457fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:39.294 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.294+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020681435 0 0) 0x7f1048136af0 con 0x7f104814be90 2024-03-25T19:44:39.294 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.294+0000 7f10457fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.294 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.294+0000 7f10457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.294 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.294+0000 7f10457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.294 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.294+0000 7f10457fa640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:44:39.294 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.294+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f102c004eb0 con 0x7f104814be90 2024-03-25T19:44:39.303 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.304+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868076684 0 0) 0x7f1048136c80 con 0x7f10480b4d60 2024-03-25T19:44:39.303 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.304+0000 7f10457fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.303 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.304+0000 7f10457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.304 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.304+0000 7f10457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.304 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.304+0000 7f10457fa640 10 monclient(hunting): my global_id is 4331 2024-03-25T19:44:39.304 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.304+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1048136af0 con 0x7f10480b4d60 2024-03-25T19:44:39.309 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.309+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659496286 0 0) 0x7f1048136250 con 0x7f10480b1730 2024-03-25T19:44:39.309 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.309+0000 7f10457fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:39.309 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.309+0000 7f10457fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:39.309 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.309+0000 7f10457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:39.309 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.309+0000 7f10457fa640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:44:39.309 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.309+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1048136c80 con 0x7f10480b1730 2024-03-25T19:44:39.310 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.310+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1738484413 0 0) 0x7f102c004eb0 con 0x7f104814be90 2024-03-25T19:44:39.310 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.310+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1048136250 con 0x7f104814be90 2024-03-25T19:44:39.310 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.310+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1040001c50 con 0x7f104814be90 2024-03-25T19:44:39.310 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.310+0000 7f10457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:39.310 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.310+0000 7f10457fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:39.310 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.310+0000 7f10457fa640 10 monclient(hunting): dump: 2024-03-25T19:44:39.310 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:39.310 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:39.310 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:39.310 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:39.310 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:39.310 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:39.311 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:39.311 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:39.311 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:39.311 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:39.311 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.310+0000 7f10457fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:39.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.318+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2056206939 0 0) 0x7f1048136af0 con 0x7f10480b4d60 2024-03-25T19:44:39.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.318+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f102c004eb0 con 0x7f10480b4d60 2024-03-25T19:44:39.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.318+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1034001ae0 con 0x7f10480b4d60 2024-03-25T19:44:39.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.318+0000 7f10457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:39.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.318+0000 7f10457fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:39.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.318+0000 7f10457fa640 10 monclient(hunting): dump: 2024-03-25T19:44:39.319 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:39.319 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:39.319 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:39.319 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:39.319 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:39.319 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:39.319 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2610539334 0 0) 0x7f1048136250 con 0x7f104814be90 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 >> v1:172.21.15.81:6790/0 conn(0x7f10480b4d60 legacy=0x7f104812fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 >> v1:172.21.15.81:6789/0 conn(0x7f10480b1730 legacy=0x7f10481d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f10457fa640 10 monclient: _finish_hunting 0 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f10457fa640 1 monclient: found mon.b 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f10457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f10457fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1048138210 con 0x7f104814be90 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f10457fa640 10 monclient: _finish_auth 0 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f10457fa640 10 monclient: _check_auth_tickets 2024-03-25T19:44:39.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f10457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:39.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f104eb8d640 5 monclient: authenticate success, global_id 4345 2024-03-25T19:44:39.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f104eb8d640 10 monclient: _renew_subs 2024-03-25T19:44:39.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f104eb8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:44:39.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f104eb8d640 1 -- 172.21.15.81:0/1929282703 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10481384c0 con 0x7f104814be90 2024-03-25T19:44:39.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f104eb8d640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:39.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f104eb8d640 10 monclient: _renew_subs 2024-03-25T19:44:39.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f104eb8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:44:39.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f104eb8d640 1 -- 172.21.15.81:0/1929282703 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1048145700 con 0x7f104814be90 2024-03-25T19:44:39.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.319+0000 7f104eb8d640 1 librados: init done 2024-03-25T19:44:39.322 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.320+0000 7f104eb8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:39.322 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.320+0000 7f104eb8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:39.322 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.320+0000 7f104eb8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:44:39.322 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.320+0000 7f104eb8d640 1 -- 172.21.15.81:0/1929282703 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10481360a0 con 0x7f104814be90 2024-03-25T19:44:39.588 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.588+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1040004b60 con 0x7f104814be90 2024-03-25T19:44:39.588 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.588+0000 7f10457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:39.588 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.588+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1040004e70 con 0x7f104814be90 2024-03-25T19:44:39.588 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.588+0000 7f10457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:39.588 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.588+0000 7f10457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:39.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.588+0000 7f10457fa640 10 monclient: dump: 2024-03-25T19:44:39.589 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:39.589 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:39.589 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:39.589 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:39.589 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:39.589 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:39.589 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:39.589 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:39.589 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:39.589 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:39.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.588+0000 7f1046ffd640 4 set_mon_vals no callback set 2024-03-25T19:44:39.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.589+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f104001ae40 con 0x7f104814be90 2024-03-25T19:44:39.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.590+0000 7f104c929640 10 monclient: get_auth_request con 0x7f102c06df20 auth_method 0 2024-03-25T19:44:39.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.590+0000 7f10457fa640 10 client.4345.objecter ms_handle_connect 0x7f102c06df20 2024-03-25T19:44:39.594 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.594+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f10400549c0 con 0x7f104814be90 2024-03-25T19:44:39.594 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.594+0000 7f10457fa640 10 client.4345.objecter ms_dispatch 0x7f1048000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:39.594 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.594+0000 7f10457fa640 3 client.4345.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:39.594 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.594+0000 7f10457fa640 3 client.4345.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:39.594 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.594+0000 7f10457fa640 20 client.4345.objecter dump_active .. 0 homeless 2024-03-25T19:44:39.595 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.595+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1040050300 con 0x7f104814be90 2024-03-25T19:44:39.595 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.595+0000 7f10457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:39.595 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.595+0000 7f10457fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:39.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.831+0000 7f104eb8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:39.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.831+0000 7f104eb8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:39.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.831+0000 7f104eb8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:44:39.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.831+0000 7f104eb8d640 1 -- 172.21.15.81:0/1929282703 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1048135ec0 con 0x7f104814be90 2024-03-25T19:44:39.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.862+0000 7f10457fa640 1 -- 172.21.15.81:0/1929282703 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f1040052400 con 0x7f104814be90 2024-03-25T19:44:39.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.863+0000 7f10457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:39.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.863+0000 7f10457fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:39.862 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:44:39.863 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:39.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.865+0000 7f104eb8d640 10 librados: watch_flush enter 2024-03-25T19:44:39.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.865+0000 7f104eb8d640 10 librados: watch_flush exit 2024-03-25T19:44:39.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.865+0000 7f104eb8d640 20 client.4345.objecter shutdown clearing up homeless session... 2024-03-25T19:44:39.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.865+0000 7f104eb8d640 10 client.4345.objecter successfully canceled tick 2024-03-25T19:44:39.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.865+0000 7f104eb8d640 1 -- 172.21.15.81:0/1929282703 >> v1:172.21.15.98:6800/3980829775 conn(0x7f102c06df20 legacy=0x7f102c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.865+0000 7f104eb8d640 10 monclient: shutdown 2024-03-25T19:44:39.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.865+0000 7f104eb8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:39.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.865+0000 7f104eb8d640 1 -- 172.21.15.81:0/1929282703 >> v1:172.21.15.98:6789/0 conn(0x7f104814be90 legacy=0x7f10481e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.865+0000 7f104eb8d640 1 -- 172.21.15.81:0/1929282703 shutdown_connections 2024-03-25T19:44:39.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.865+0000 7f104eb8d640 1 -- 172.21.15.81:0/1929282703 >> 172.21.15.81:0/1929282703 conn(0x7f10480617a0 msgr2=0x7f10480b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:39.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.865+0000 7f104eb8d640 1 -- 172.21.15.81:0/1929282703 shutdown_connections 2024-03-25T19:44:39.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.865+0000 7f104eb8d640 1 -- 172.21.15.81:0/1929282703 wait complete. 2024-03-25T19:44:39.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:39.865+0000 7f104eb8d640 1 librados: shutdown 2024-03-25T19:44:39.876 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:46.683 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:46.798 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.799+0000 7f992b686640 10 monclient: get_monmap_and_config 2024-03-25T19:44:46.799 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.799+0000 7f992b686640 10 monclient: build_initial_monmap 2024-03-25T19:44:46.799 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.799+0000 7f992b686640 10 monclient: monmap: 2024-03-25T19:44:46.799 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:44:46.799 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:46.799 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:44:46.800267+0000 2024-03-25T19:44:46.799 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:44:46.800267+0000 2024-03-25T19:44:46.799 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:46.799 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:46.799 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:44:46.799 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:44:46.800 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:44:46.800 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:46.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.799+0000 7f992b686640 1 Processor -- start 2024-03-25T19:44:46.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.800+0000 7f992b686640 1 -- start start 2024-03-25T19:44:46.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.800+0000 7f992b686640 10 monclient: init 2024-03-25T19:44:46.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.800+0000 7f992b686640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:46.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.800+0000 7f992b686640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:46.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.800+0000 7f992b686640 10 monclient(hunting): picked mon.noname-a con 0x7f99240a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:44:46.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.800+0000 7f992b686640 10 monclient(hunting): picked mon.noname-c con 0x7f992413f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:44:46.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.800+0000 7f992b686640 10 monclient(hunting): picked mon.noname-b con 0x7f99241430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:44:46.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.800+0000 7f992b686640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9924142520 con 0x7f99240a98f0 2024-03-25T19:44:46.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.800+0000 7f992b686640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99240b28a0 con 0x7f992413f710 2024-03-25T19:44:46.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.800+0000 7f992b686640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99240b2a30 con 0x7f99241430c0 2024-03-25T19:44:46.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.800+0000 7f992b686640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:46.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.800+0000 7f992b686640 10 monclient(hunting): authenticate will time out at 4121.882714s 2024-03-25T19:44:46.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.809+0000 7f9929c23640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f99241430c0 0x7f992414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:40286/0 (socket says 172.21.15.81:40286) 2024-03-25T19:44:46.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.809+0000 7f9929c23640 1 -- 172.21.15.81:0/622792259 learned_addr learned my addr 172.21.15.81:0/622792259 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:44:46.826 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.826+0000 7f991bfff640 1 -- 172.21.15.81:0/622792259 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693353342 0 0) 0x7f99240b28a0 con 0x7f992413f710 2024-03-25T19:44:46.826 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.826+0000 7f991bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:46.826 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.826+0000 7f991bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:46.826 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.826+0000 7f991bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:46.826 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.826+0000 7f991bfff640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:44:46.826 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.826+0000 7f991bfff640 1 -- 172.21.15.81:0/622792259 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9910003520 con 0x7f992413f710 2024-03-25T19:44:46.834 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.835+0000 7f991bfff640 1 -- 172.21.15.81:0/622792259 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552524954 0 0) 0x7f99240b2a30 con 0x7f99241430c0 2024-03-25T19:44:46.834 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.835+0000 7f991bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:46.834 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.835+0000 7f991bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:46.834 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.835+0000 7f991bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:46.834 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.835+0000 7f991bfff640 10 monclient(hunting): my global_id is 4334 2024-03-25T19:44:46.834 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.835+0000 7f991bfff640 1 -- 172.21.15.81:0/622792259 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99240b28a0 con 0x7f99241430c0 2024-03-25T19:44:46.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.845+0000 7f991bfff640 1 -- 172.21.15.81:0/622792259 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3289906444 0 0) 0x7f99240b28a0 con 0x7f99241430c0 2024-03-25T19:44:46.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.845+0000 7f991bfff640 1 -- 172.21.15.81:0/622792259 >> v1:172.21.15.98:6789/0 conn(0x7f992413f710 legacy=0x7f99241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.845+0000 7f991bfff640 1 -- 172.21.15.81:0/622792259 >> v1:172.21.15.81:6789/0 conn(0x7f99240a98f0 legacy=0x7f99240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.845+0000 7f991bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:46.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.845+0000 7f991bfff640 1 monclient: found mon.noname-b 2024-03-25T19:44:46.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.845+0000 7f991bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:46.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.845+0000 7f991bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:44:46.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.845+0000 7f991bfff640 1 -- 172.21.15.81:0/622792259 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99240b44c0 con 0x7f99241430c0 2024-03-25T19:44:46.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.845+0000 7f991bfff640 10 monclient: _finish_auth 0 2024-03-25T19:44:46.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.845+0000 7f991bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:46.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.845+0000 7f991bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:46.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.845+0000 7f991bfff640 1 -- 172.21.15.81:0/622792259 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f991c0019f0 con 0x7f99241430c0 2024-03-25T19:44:46.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.845+0000 7f991bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:46.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.845+0000 7f991bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:46.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.845+0000 7f991bfff640 10 monclient: dump: 2024-03-25T19:44:46.846 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:46.846 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:46.846 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:46.846 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:46.846 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:46.846 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:46.846 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:46.846 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:46.846 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:46.846 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:46.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.845+0000 7f992b686640 5 monclient: authenticate success, global_id 4334 2024-03-25T19:44:46.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.845+0000 7f992b686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:46.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.860+0000 7f991bfff640 1 -- 172.21.15.81:0/622792259 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f991c001b50 con 0x7f99241430c0 2024-03-25T19:44:46.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.860+0000 7f991bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:46.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.860+0000 7f991bfff640 1 -- 172.21.15.81:0/622792259 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f991c001fa0 con 0x7f99241430c0 2024-03-25T19:44:46.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.860+0000 7f991bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:46.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.860+0000 7f991bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:46.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.860+0000 7f991bfff640 10 monclient: dump: 2024-03-25T19:44:46.860 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:46.860 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:46.860 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:46.860 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:46.861 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:46.861 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:46.861 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:46.861 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:46.861 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:46.861 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:46.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.860+0000 7f992b686640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:46.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.860+0000 7f992b686640 4 set_mon_vals no callback set 2024-03-25T19:44:46.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.861+0000 7f992b686640 10 monclient: shutdown 2024-03-25T19:44:46.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.861+0000 7f992b686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:46.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.861+0000 7f992b686640 1 -- 172.21.15.81:0/622792259 >> v1:172.21.15.81:6790/0 conn(0x7f99241430c0 legacy=0x7f992414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.861+0000 7f992b686640 1 -- 172.21.15.81:0/622792259 shutdown_connections 2024-03-25T19:44:46.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.861+0000 7f992b686640 1 -- 172.21.15.81:0/622792259 >> 172.21.15.81:0/622792259 conn(0x7f99240b7ea0 msgr2=0x7f99240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:46.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.861+0000 7f992b686640 1 -- 172.21.15.81:0/622792259 shutdown_connections 2024-03-25T19:44:46.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.861+0000 7f992b686640 1 -- 172.21.15.81:0/622792259 wait complete. 2024-03-25T19:44:46.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.862+0000 7f992b686640 10 monclient: build_initial_monmap 2024-03-25T19:44:46.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.862+0000 7f992b686640 10 monclient: monmap: 2024-03-25T19:44:46.863 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:44:46.863 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:46.863 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:44:46.863 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:44:46.863 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:46.863 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:44:46.863 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:44:46.863 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:44:46.863 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:44:46.864 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:46.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.862+0000 7f992b686640 1 librados: starting msgr at 2024-03-25T19:44:46.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.862+0000 7f992b686640 1 librados: starting objecter 2024-03-25T19:44:46.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.863+0000 7f992b686640 1 Processor -- start 2024-03-25T19:44:46.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.863+0000 7f992b686640 1 -- start start 2024-03-25T19:44:46.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.863+0000 7f992b686640 1 librados: setting wanted keys 2024-03-25T19:44:46.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.863+0000 7f992b686640 1 librados: calling monclient init 2024-03-25T19:44:46.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.863+0000 7f992b686640 10 monclient: init 2024-03-25T19:44:46.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.863+0000 7f992b686640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:46.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.863+0000 7f992b686640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:46.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.863+0000 7f992b686640 10 monclient(hunting): picked mon.noname-c con 0x7f99240a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:44:46.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.863+0000 7f992b686640 10 monclient(hunting): picked mon.noname-b con 0x7f992413f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:44:46.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.864+0000 7f992b686640 10 monclient(hunting): picked mon.noname-a con 0x7f99241430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:44:46.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.864+0000 7f992b686640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9924139610 con 0x7f99241430c0 2024-03-25T19:44:46.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.864+0000 7f992b686640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992413a870 con 0x7f992413f710 2024-03-25T19:44:46.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.864+0000 7f992b686640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992413b4c0 con 0x7f99240a98f0 2024-03-25T19:44:46.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.864+0000 7f992b686640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:46.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.864+0000 7f992b686640 10 monclient(hunting): authenticate will time out at 4121.946038s 2024-03-25T19:44:46.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.876+0000 7f9929422640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f99240a98f0 0x7f99240b71c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:40338/0 (socket says 172.21.15.81:40338) 2024-03-25T19:44:46.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.876+0000 7f9929422640 1 -- 172.21.15.81:0/452279470 learned_addr learned my addr 172.21.15.81:0/452279470 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:44:46.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.878+0000 7f9919ffb640 10 client.?.objecter ms_handle_connect 0x7f99240a98f0 2024-03-25T19:44:46.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.878+0000 7f9919ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:46.879 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.880+0000 7f9919ffb640 10 client.?.objecter ms_handle_connect 0x7f992413f710 2024-03-25T19:44:46.879 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.880+0000 7f9919ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:46.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.887+0000 7f9919ffb640 10 client.?.objecter ms_handle_connect 0x7f99241430c0 2024-03-25T19:44:46.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.887+0000 7f9919ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:46.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.896+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658369130 0 0) 0x7f9924139610 con 0x7f99241430c0 2024-03-25T19:44:46.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.896+0000 7f9919ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:46.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.896+0000 7f9919ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:46.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.896+0000 7f9919ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:46.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.896+0000 7f9919ffb640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:44:46.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.896+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9908004eb0 con 0x7f99241430c0 2024-03-25T19:44:46.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.897+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976723067 0 0) 0x7f992413b4c0 con 0x7f99240a98f0 2024-03-25T19:44:46.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.897+0000 7f9919ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:46.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.897+0000 7f9919ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:46.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.897+0000 7f9919ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:46.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.897+0000 7f9919ffb640 10 monclient(hunting): my global_id is 4337 2024-03-25T19:44:46.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.897+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9924139610 con 0x7f99240a98f0 2024-03-25T19:44:46.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.898+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606387844 0 0) 0x7f992413a870 con 0x7f992413f710 2024-03-25T19:44:46.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.898+0000 7f9919ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:46.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.898+0000 7f9919ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:46.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.898+0000 7f9919ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:46.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.898+0000 7f9919ffb640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:44:46.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.898+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f992413b4c0 con 0x7f992413f710 2024-03-25T19:44:46.915 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.915+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3903895004 0 0) 0x7f9924139610 con 0x7f99240a98f0 2024-03-25T19:44:46.915 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.915+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f992413a870 con 0x7f99240a98f0 2024-03-25T19:44:46.915 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.915+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f99200019b0 con 0x7f99240a98f0 2024-03-25T19:44:46.915 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.915+0000 7f9919ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:46.915 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.915+0000 7f9919ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:46.915 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.915+0000 7f9919ffb640 10 monclient(hunting): dump: 2024-03-25T19:44:46.915 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:46.915 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:46.915 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:46.915 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:46.915 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:46.915 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:46.915 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:46.916 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:46.916 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:46.916 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:46.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.915+0000 7f9919ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:46.924 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.924+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3314334601 0 0) 0x7f992413b4c0 con 0x7f992413f710 2024-03-25T19:44:46.924 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.924+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9924139610 con 0x7f992413f710 2024-03-25T19:44:46.924 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.924+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f99140017b0 con 0x7f992413f710 2024-03-25T19:44:46.924 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.924+0000 7f9919ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:46.924 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.924+0000 7f9919ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:46.924 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.924+0000 7f9919ffb640 10 monclient(hunting): dump: 2024-03-25T19:44:46.924 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:46.924 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:46.924 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:46.924 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:46.924 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:46.924 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:46.924 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:46.924 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:46.925 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:46.925 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:46.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.932+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2837641641 0 0) 0x7f9908004eb0 con 0x7f99241430c0 2024-03-25T19:44:46.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.932+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f992413b4c0 con 0x7f99241430c0 2024-03-25T19:44:46.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.932+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f991c001e60 con 0x7f99241430c0 2024-03-25T19:44:46.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.932+0000 7f9919ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:46.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.932+0000 7f9919ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:46.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.932+0000 7f9919ffb640 10 monclient(hunting): dump: 2024-03-25T19:44:46.932 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:46.932 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:46.932 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:46.932 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:46.932 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:46.932 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:46.932 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:46.932 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:46.933 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:46.933 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:46.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1647557984 0 0) 0x7f992413a870 con 0x7f99240a98f0 2024-03-25T19:44:46.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 >> v1:172.21.15.98:6789/0 conn(0x7f992413f710 legacy=0x7f99241dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 >> v1:172.21.15.81:6789/0 conn(0x7f99241430c0 legacy=0x7f99241db280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f9919ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:44:46.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f9919ffb640 1 monclient: found mon.c 2024-03-25T19:44:46.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f9919ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:46.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f9919ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:44:46.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99241db990 con 0x7f99240a98f0 2024-03-25T19:44:46.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f9919ffb640 10 monclient: _finish_auth 0 2024-03-25T19:44:46.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f9919ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:44:46.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f9919ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:46.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f992b686640 5 monclient: authenticate success, global_id 4337 2024-03-25T19:44:46.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f992b686640 10 monclient: _renew_subs 2024-03-25T19:44:46.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f992b686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:44:46.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f992b686640 1 -- 172.21.15.81:0/452279470 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992413b0a0 con 0x7f99240a98f0 2024-03-25T19:44:46.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f992b686640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:46.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f992b686640 10 monclient: _renew_subs 2024-03-25T19:44:46.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f992b686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:44:46.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f992b686640 1 -- 172.21.15.81:0/452279470 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9924139d20 con 0x7f99240a98f0 2024-03-25T19:44:46.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.935+0000 7f992b686640 1 librados: init done 2024-03-25T19:44:46.937 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.936+0000 7f992b686640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:46.937 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.936+0000 7f992b686640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:46.937 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.936+0000 7f992b686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:44:46.937 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:46.936+0000 7f992b686640 1 -- 172.21.15.81:0/452279470 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f992414f2d0 con 0x7f99240a98f0 2024-03-25T19:44:47.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.221+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9920003610 con 0x7f99240a98f0 2024-03-25T19:44:47.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.221+0000 7f9919ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:47.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.221+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f99200039a0 con 0x7f99240a98f0 2024-03-25T19:44:47.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.221+0000 7f9919ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:47.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.221+0000 7f9919ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:47.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.221+0000 7f9919ffb640 10 monclient: dump: 2024-03-25T19:44:47.221 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:47.221 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:47.221 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:47.221 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:47.221 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:47.221 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:47.222 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:47.222 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:47.222 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:47.222 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:47.222 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.221+0000 7f991b7fe640 4 set_mon_vals no callback set 2024-03-25T19:44:47.222 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.221+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f9920003bc0 con 0x7f99240a98f0 2024-03-25T19:44:47.222 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.222+0000 7f9928c21640 10 monclient: get_auth_request con 0x7f990806e0b0 auth_method 0 2024-03-25T19:44:47.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.222+0000 7f9919ffb640 10 client.4337.objecter ms_handle_connect 0x7f990806e0b0 2024-03-25T19:44:47.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.226+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f9920054810 con 0x7f99240a98f0 2024-03-25T19:44:47.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.226+0000 7f9919ffb640 10 client.4337.objecter ms_dispatch 0x7f9924000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:47.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.226+0000 7f9919ffb640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:47.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.226+0000 7f9919ffb640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:47.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.226+0000 7f9919ffb640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-25T19:44:47.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.226+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9920050320 con 0x7f99240a98f0 2024-03-25T19:44:47.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.226+0000 7f9919ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:47.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.226+0000 7f9919ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:47.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.471+0000 7f992b686640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:47.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.471+0000 7f992b686640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:47.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.471+0000 7f992b686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:44:47.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.471+0000 7f992b686640 1 -- 172.21.15.81:0/452279470 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99240627b0 con 0x7f99240a98f0 2024-03-25T19:44:47.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.504+0000 7f9919ffb640 1 -- 172.21.15.81:0/452279470 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f9920052250 con 0x7f99240a98f0 2024-03-25T19:44:47.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.504+0000 7f9919ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:47.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.504+0000 7f9919ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:47.504 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:44:47.504 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:47.506 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.506+0000 7f992b686640 10 librados: watch_flush enter 2024-03-25T19:44:47.506 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.506+0000 7f992b686640 10 librados: watch_flush exit 2024-03-25T19:44:47.506 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.506+0000 7f992b686640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-25T19:44:47.507 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.506+0000 7f992b686640 10 client.4337.objecter successfully canceled tick 2024-03-25T19:44:47.507 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.506+0000 7f992b686640 1 -- 172.21.15.81:0/452279470 >> v1:172.21.15.98:6800/3980829775 conn(0x7f990806e0b0 legacy=0x7f9908070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:47.507 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.506+0000 7f992b686640 10 monclient: shutdown 2024-03-25T19:44:47.507 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.506+0000 7f992b686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:47.507 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.506+0000 7f992b686640 1 -- 172.21.15.81:0/452279470 >> v1:172.21.15.81:6790/0 conn(0x7f99240a98f0 legacy=0x7f99240b71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:47.507 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.506+0000 7f992b686640 1 -- 172.21.15.81:0/452279470 shutdown_connections 2024-03-25T19:44:47.507 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.506+0000 7f992b686640 1 -- 172.21.15.81:0/452279470 >> 172.21.15.81:0/452279470 conn(0x7f99240b7ea0 msgr2=0x7f9924074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:47.507 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.506+0000 7f992b686640 1 -- 172.21.15.81:0/452279470 shutdown_connections 2024-03-25T19:44:47.507 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.506+0000 7f992b686640 1 -- 172.21.15.81:0/452279470 wait complete. 2024-03-25T19:44:47.507 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:47.506+0000 7f992b686640 1 librados: shutdown 2024-03-25T19:44:47.516 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:54.323 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:54.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.442+0000 7fdb0b4f1640 10 monclient: get_monmap_and_config 2024-03-25T19:44:54.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.442+0000 7fdb0b4f1640 10 monclient: build_initial_monmap 2024-03-25T19:44:54.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.442+0000 7fdb0b4f1640 10 monclient: monmap: 2024-03-25T19:44:54.442 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:44:54.442 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:54.442 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:44:54.443328+0000 2024-03-25T19:44:54.442 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:44:54.443328+0000 2024-03-25T19:44:54.442 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:54.442 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:54.442 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:44:54.442 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:44:54.443 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:44:54.443 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:54.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.443+0000 7fdb0b4f1640 1 Processor -- start 2024-03-25T19:44:54.444 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.443+0000 7fdb0b4f1640 1 -- start start 2024-03-25T19:44:54.444 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.443+0000 7fdb0b4f1640 10 monclient: init 2024-03-25T19:44:54.444 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.444+0000 7fdb0b4f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:54.444 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.444+0000 7fdb0b4f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:54.444 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.444+0000 7fdb0b4f1640 10 monclient(hunting): picked mon.noname-a con 0x7fdb040b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:44:54.444 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.444+0000 7fdb0b4f1640 10 monclient(hunting): picked mon.noname-b con 0x7fdb040b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:44:54.444 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.444+0000 7fdb0b4f1640 10 monclient(hunting): picked mon.noname-c con 0x7fdb0414be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:44:54.444 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.444+0000 7fdb0b4f1640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb040af940 con 0x7fdb040b4d60 2024-03-25T19:44:54.444 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.444+0000 7fdb0b4f1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb040b0a70 con 0x7fdb0414be90 2024-03-25T19:44:54.444 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.444+0000 7fdb0b4f1640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb040b0d50 con 0x7fdb040b1730 2024-03-25T19:44:54.444 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.444+0000 7fdb0b4f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:54.444 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.444+0000 7fdb0b4f1640 10 monclient(hunting): authenticate will time out at 4129.526194s 2024-03-25T19:44:54.445 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.446+0000 7fdb0928d640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fdb040b4d60 0x7fdb0413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:57104/0 (socket says 172.21.15.81:57104) 2024-03-25T19:44:54.445 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.446+0000 7fdb0928d640 1 -- 172.21.15.81:0/1155668991 learned_addr learned my addr 172.21.15.81:0/1155668991 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:44:54.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.454+0000 7fdafbfff640 1 -- 172.21.15.81:0/1155668991 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851322121 0 0) 0x7fdb040af940 con 0x7fdb040b4d60 2024-03-25T19:44:54.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.454+0000 7fdafbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:54.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.454+0000 7fdafbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:54.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.454+0000 7fdafbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:54.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.454+0000 7fdafbfff640 10 monclient(hunting): my global_id is 4356 2024-03-25T19:44:54.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.454+0000 7fdafbfff640 1 -- 172.21.15.81:0/1155668991 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaf0003520 con 0x7fdb040b4d60 2024-03-25T19:44:54.460 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.461+0000 7fdafbfff640 1 -- 172.21.15.81:0/1155668991 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055477389 0 0) 0x7fdb040b0a70 con 0x7fdb0414be90 2024-03-25T19:44:54.461 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.461+0000 7fdafbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:54.461 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.461+0000 7fdafbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:54.461 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.461+0000 7fdafbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:54.461 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.461+0000 7fdafbfff640 10 monclient(hunting): my global_id is 4354 2024-03-25T19:44:54.461 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.461+0000 7fdafbfff640 1 -- 172.21.15.81:0/1155668991 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb040af940 con 0x7fdb0414be90 2024-03-25T19:44:54.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.464+0000 7fdafbfff640 1 -- 172.21.15.81:0/1155668991 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1082286156 0 0) 0x7fdaf0003520 con 0x7fdb040b4d60 2024-03-25T19:44:54.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.464+0000 7fdafbfff640 1 -- 172.21.15.81:0/1155668991 >> v1:172.21.15.81:6790/0 conn(0x7fdb040b1730 legacy=0x7fdb040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.464+0000 7fdafbfff640 1 -- 172.21.15.81:0/1155668991 >> v1:172.21.15.98:6789/0 conn(0x7fdb0414be90 legacy=0x7fdb0414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.464+0000 7fdafbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:54.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.464+0000 7fdafbfff640 1 monclient: found mon.noname-a 2024-03-25T19:44:54.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.464+0000 7fdafbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:54.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.464+0000 7fdafbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:44:54.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.464+0000 7fdafbfff640 1 -- 172.21.15.81:0/1155668991 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb040b1030 con 0x7fdb040b4d60 2024-03-25T19:44:54.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.464+0000 7fdafbfff640 10 monclient: _finish_auth 0 2024-03-25T19:44:54.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.464+0000 7fdafbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:54.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.464+0000 7fdafbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:54.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.464+0000 7fdafbfff640 1 -- 172.21.15.81:0/1155668991 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fdb00001690 con 0x7fdb040b4d60 2024-03-25T19:44:54.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.464+0000 7fdb0b4f1640 5 monclient: authenticate success, global_id 4356 2024-03-25T19:44:54.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.464+0000 7fdb0b4f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:54.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.464+0000 7fdafbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:54.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.464+0000 7fdafbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:54.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.464+0000 7fdafbfff640 10 monclient: dump: 2024-03-25T19:44:54.465 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:54.466 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:54.466 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:54.466 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:54.466 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:54.466 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:54.466 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:54.466 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:54.466 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:54.466 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:54.466 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.465+0000 7fdb0b4f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:54.472 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.472+0000 7fdafbfff640 1 -- 172.21.15.81:0/1155668991 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb000017f0 con 0x7fdb040b4d60 2024-03-25T19:44:54.472 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.472+0000 7fdafbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:54.472 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.472+0000 7fdb0b4f1640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:54.472 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.472+0000 7fdb0b4f1640 4 set_mon_vals no callback set 2024-03-25T19:44:54.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.473+0000 7fdb0b4f1640 10 monclient: shutdown 2024-03-25T19:44:54.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.473+0000 7fdb0b4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:54.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.473+0000 7fdb0b4f1640 1 -- 172.21.15.81:0/1155668991 >> v1:172.21.15.81:6789/0 conn(0x7fdb040b4d60 legacy=0x7fdb0413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.473+0000 7fdb0b4f1640 1 -- 172.21.15.81:0/1155668991 shutdown_connections 2024-03-25T19:44:54.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.473+0000 7fdb0b4f1640 1 -- 172.21.15.81:0/1155668991 >> 172.21.15.81:0/1155668991 conn(0x7fdb040617a0 msgr2=0x7fdb04061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:54.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.473+0000 7fdb0b4f1640 1 -- 172.21.15.81:0/1155668991 shutdown_connections 2024-03-25T19:44:54.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.473+0000 7fdb0b4f1640 1 -- 172.21.15.81:0/1155668991 wait complete. 2024-03-25T19:44:54.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.474+0000 7fdb0b4f1640 10 monclient: build_initial_monmap 2024-03-25T19:44:54.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.474+0000 7fdb0b4f1640 10 monclient: monmap: 2024-03-25T19:44:54.474 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:44:54.475 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:54.475 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:44:54.475 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:44:54.475 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:54.475 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:44:54.475 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:44:54.475 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:44:54.475 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:44:54.475 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:54.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.475+0000 7fdb0b4f1640 1 librados: starting msgr at 2024-03-25T19:44:54.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.475+0000 7fdb0b4f1640 1 librados: starting objecter 2024-03-25T19:44:54.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.475+0000 7fdb0b4f1640 1 Processor -- start 2024-03-25T19:44:54.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.475+0000 7fdb0b4f1640 1 -- start start 2024-03-25T19:44:54.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.475+0000 7fdb0b4f1640 1 librados: setting wanted keys 2024-03-25T19:44:54.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.475+0000 7fdb0b4f1640 1 librados: calling monclient init 2024-03-25T19:44:54.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.475+0000 7fdb0b4f1640 10 monclient: init 2024-03-25T19:44:54.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.475+0000 7fdb0b4f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:54.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.475+0000 7fdb0b4f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:54.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.475+0000 7fdb0b4f1640 10 monclient(hunting): picked mon.noname-b con 0x7fdb040b1730 addr v1:172.21.15.98:6789/0 2024-03-25T19:44:54.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.475+0000 7fdb0b4f1640 10 monclient(hunting): picked mon.noname-a con 0x7fdb040b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:44:54.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.475+0000 7fdb0b4f1640 10 monclient(hunting): picked mon.noname-c con 0x7fdb0414be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:44:54.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.475+0000 7fdb0b4f1640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb041d7590 con 0x7fdb040b4d60 2024-03-25T19:44:54.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.475+0000 7fdb0b4f1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb04140380 con 0x7fdb040b1730 2024-03-25T19:44:54.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.476+0000 7fdb0b4f1640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb04140830 con 0x7fdb0414be90 2024-03-25T19:44:54.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.476+0000 7fdb0b4f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:54.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.476+0000 7fdb0b4f1640 10 monclient(hunting): authenticate will time out at 4129.557857s 2024-03-25T19:44:54.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.481+0000 7fdb08a8c640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fdb040b4d60 0x7fdb0412fef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:57136/0 (socket says 172.21.15.81:57136) 2024-03-25T19:44:54.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.481+0000 7fdb08a8c640 1 -- 172.21.15.81:0/3686992013 learned_addr learned my addr 172.21.15.81:0/3686992013 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:44:54.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.482+0000 7fdaf9ffb640 10 client.?.objecter ms_handle_connect 0x7fdb040b4d60 2024-03-25T19:44:54.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.482+0000 7fdaf9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:54.487 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.488+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554927054 0 0) 0x7fdb041d7590 con 0x7fdb040b4d60 2024-03-25T19:44:54.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.488+0000 7fdaf9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:54.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.488+0000 7fdaf9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:54.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.488+0000 7fdaf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:54.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.488+0000 7fdaf9ffb640 10 monclient(hunting): my global_id is 4359 2024-03-25T19:44:54.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.488+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdae8004eb0 con 0x7fdb040b4d60 2024-03-25T19:44:54.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.490+0000 7fdaf9ffb640 10 client.?.objecter ms_handle_connect 0x7fdb0414be90 2024-03-25T19:44:54.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.490+0000 7fdaf9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:54.491 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.492+0000 7fdaf9ffb640 10 client.?.objecter ms_handle_connect 0x7fdb040b1730 2024-03-25T19:44:54.491 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.492+0000 7fdaf9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:54.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.498+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1242292482 0 0) 0x7fdae8004eb0 con 0x7fdb040b4d60 2024-03-25T19:44:54.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.498+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb041d7590 con 0x7fdb040b4d60 2024-03-25T19:44:54.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.498+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fdaf4001a30 con 0x7fdb040b4d60 2024-03-25T19:44:54.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.498+0000 7fdaf9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:54.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.498+0000 7fdaf9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:54.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.498+0000 7fdaf9ffb640 10 monclient(hunting): dump: 2024-03-25T19:44:54.498 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:54.498 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:54.498 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:54.498 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:54.498 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:54.499 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:54.499 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:54.499 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:54.499 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:54.499 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:54.499 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.498+0000 7fdaf9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:54.499 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.499+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352996523 0 0) 0x7fdb04140380 con 0x7fdb040b1730 2024-03-25T19:44:54.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.499+0000 7fdaf9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:54.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.499+0000 7fdaf9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:54.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.499+0000 7fdaf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:54.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.499+0000 7fdaf9ffb640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:44:54.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.499+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdae8004eb0 con 0x7fdb040b1730 2024-03-25T19:44:54.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.504+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185484300 0 0) 0x7fdb04140830 con 0x7fdb0414be90 2024-03-25T19:44:54.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.504+0000 7fdaf9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:54.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.504+0000 7fdaf9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:54.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.504+0000 7fdaf9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:54.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.504+0000 7fdaf9ffb640 10 monclient(hunting): my global_id is 4343 2024-03-25T19:44:54.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.504+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb04140380 con 0x7fdb0414be90 2024-03-25T19:44:54.508 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2813724452 0 0) 0x7fdb041d7590 con 0x7fdb040b4d60 2024-03-25T19:44:54.508 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 >> v1:172.21.15.81:6790/0 conn(0x7fdb0414be90 legacy=0x7fdb041d7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.508 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 >> v1:172.21.15.98:6789/0 conn(0x7fdb040b1730 legacy=0x7fdb04141280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.508 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdaf9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:44:54.508 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdaf9ffb640 1 monclient: found mon.a 2024-03-25T19:44:54.508 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdaf9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:54.508 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdaf9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:44:54.508 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb04143900 con 0x7fdb040b4d60 2024-03-25T19:44:54.508 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdaf9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:44:54.508 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdaf9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:44:54.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdaf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:54.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdb0b4f1640 5 monclient: authenticate success, global_id 4359 2024-03-25T19:44:54.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdb0b4f1640 10 monclient: _renew_subs 2024-03-25T19:44:54.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdb0b4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:44:54.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdb0b4f1640 1 -- 172.21.15.81:0/3686992013 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb04141c00 con 0x7fdb040b4d60 2024-03-25T19:44:54.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdb0b4f1640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:54.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdb0b4f1640 10 monclient: _renew_subs 2024-03-25T19:44:54.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdb0b4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:44:54.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdb0b4f1640 1 -- 172.21.15.81:0/3686992013 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb041307e0 con 0x7fdb040b4d60 2024-03-25T19:44:54.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.508+0000 7fdb0b4f1640 1 librados: init done 2024-03-25T19:44:54.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.509+0000 7fdb0b4f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:54.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.509+0000 7fdb0b4f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:54.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.509+0000 7fdb0b4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:44:54.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.509+0000 7fdb0b4f1640 1 -- 172.21.15.81:0/3686992013 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb04142d00 con 0x7fdb040b4d60 2024-03-25T19:44:54.799 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.799+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdaf4001cb0 con 0x7fdb040b4d60 2024-03-25T19:44:54.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.800+0000 7fdaf9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:54.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.800+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fdaf4003a60 con 0x7fdb040b4d60 2024-03-25T19:44:54.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.800+0000 7fdaf9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:54.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.800+0000 7fdaf9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:54.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.800+0000 7fdaf9ffb640 10 monclient: dump: 2024-03-25T19:44:54.800 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:44:54.800 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:44:54.800 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:54.800 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:44:54.800 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:54.800 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:44:54.800 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:44:54.801 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:44:54.801 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:44:54.801 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:44:54.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.800+0000 7fdafb7fe640 4 set_mon_vals no callback set 2024-03-25T19:44:54.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.800+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fdaf4003c80 con 0x7fdb040b4d60 2024-03-25T19:44:54.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.801+0000 7fdb0928d640 10 monclient: get_auth_request con 0x7fdae806de50 auth_method 0 2024-03-25T19:44:54.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.802+0000 7fdaf9ffb640 10 client.4359.objecter ms_handle_connect 0x7fdae806de50 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.807+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fdaf4054910 con 0x7fdb040b4d60 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.807+0000 7fdaf9ffb640 10 client.4359.objecter ms_dispatch 0x7fdb04000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.807+0000 7fdaf9ffb640 3 client.4359.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.807+0000 7fdaf9ffb640 3 client.4359.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.807+0000 7fdaf9ffb640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.807+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdaf4050250 con 0x7fdb040b4d60 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.807+0000 7fdaf9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:54.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:54.807+0000 7fdaf9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:55.058 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.059+0000 7fdb0b4f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:55.058 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.059+0000 7fdb0b4f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:55.058 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.059+0000 7fdb0b4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:44:55.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.059+0000 7fdb0b4f1640 1 -- 172.21.15.81:0/3686992013 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb04141990 con 0x7fdb040b4d60 2024-03-25T19:44:55.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.083+0000 7fdaf9ffb640 1 -- 172.21.15.81:0/3686992013 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fdaf4052350 con 0x7fdb040b4d60 2024-03-25T19:44:55.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.083+0000 7fdaf9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:55.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.083+0000 7fdaf9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:55.083 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:44:55.083 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:55.085 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.085+0000 7fdb0b4f1640 10 librados: watch_flush enter 2024-03-25T19:44:55.085 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.085+0000 7fdb0b4f1640 10 librados: watch_flush exit 2024-03-25T19:44:55.085 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.085+0000 7fdb0b4f1640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-25T19:44:55.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.085+0000 7fdb0b4f1640 10 client.4359.objecter successfully canceled tick 2024-03-25T19:44:55.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.085+0000 7fdb0b4f1640 1 -- 172.21.15.81:0/3686992013 >> v1:172.21.15.98:6800/3980829775 conn(0x7fdae806de50 legacy=0x7fdae8070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:55.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.085+0000 7fdb0b4f1640 10 monclient: shutdown 2024-03-25T19:44:55.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.085+0000 7fdb0b4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:55.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.085+0000 7fdb0b4f1640 1 -- 172.21.15.81:0/3686992013 >> v1:172.21.15.81:6789/0 conn(0x7fdb040b4d60 legacy=0x7fdb0412fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:55.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.085+0000 7fdb0b4f1640 1 -- 172.21.15.81:0/3686992013 shutdown_connections 2024-03-25T19:44:55.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.085+0000 7fdb0b4f1640 1 -- 172.21.15.81:0/3686992013 >> 172.21.15.81:0/3686992013 conn(0x7fdb040617a0 msgr2=0x7fdb0413c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:55.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.085+0000 7fdb0b4f1640 1 -- 172.21.15.81:0/3686992013 shutdown_connections 2024-03-25T19:44:55.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.085+0000 7fdb0b4f1640 1 -- 172.21.15.81:0/3686992013 wait complete. 2024-03-25T19:44:55.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:44:55.085+0000 7fdb0b4f1640 1 librados: shutdown 2024-03-25T19:44:55.096 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:57.017 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:03:19.659 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:57.023 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:03:19.665 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:57.093 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:03:19.735 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:57.126 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:19.760 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:57.143 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:19.778 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:57.170 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:19.805 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:57.180 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:03:19.822 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:57.192 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:03:19.834 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:57.219 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:03:19.861 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:01.903 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:02.012 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.012+0000 7f9b9715d640 10 monclient: get_monmap_and_config 2024-03-25T19:45:02.012 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.012+0000 7f9b9715d640 10 monclient: build_initial_monmap 2024-03-25T19:45:02.012 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.012+0000 7f9b9715d640 10 monclient: monmap: 2024-03-25T19:45:02.012 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:45:02.012 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:02.012 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:45:02.013532+0000 2024-03-25T19:45:02.012 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:45:02.013532+0000 2024-03-25T19:45:02.012 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:02.013 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:02.013 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:45:02.013 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:45:02.013 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:45:02.013 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:02.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.013+0000 7f9b9715d640 1 Processor -- start 2024-03-25T19:45:02.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.013+0000 7f9b9715d640 1 -- start start 2024-03-25T19:45:02.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.013+0000 7f9b9715d640 10 monclient: init 2024-03-25T19:45:02.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.014+0000 7f9b9715d640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:02.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.014+0000 7f9b9715d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:02.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.014+0000 7f9b9715d640 10 monclient(hunting): picked mon.noname-c con 0x7f9b900a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:45:02.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.014+0000 7f9b9715d640 10 monclient(hunting): picked mon.noname-b con 0x7f9b9013f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:45:02.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.014+0000 7f9b9715d640 10 monclient(hunting): picked mon.noname-a con 0x7f9b901430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:45:02.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.014+0000 7f9b9715d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b90142520 con 0x7f9b901430c0 2024-03-25T19:45:02.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.014+0000 7f9b9715d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b900b28a0 con 0x7f9b900a98f0 2024-03-25T19:45:02.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.014+0000 7f9b9715d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b900b2a30 con 0x7f9b9013f710 2024-03-25T19:45:02.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.014+0000 7f9b9715d640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:02.015 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.014+0000 7f9b9715d640 10 monclient(hunting): authenticate will time out at 4137.096001s 2024-03-25T19:45:02.015 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.016+0000 7f9b956fa640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f9b901430c0 0x7f9b9014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:42200/0 (socket says 172.21.15.81:42200) 2024-03-25T19:45:02.015 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.016+0000 7f9b956fa640 1 -- 172.21.15.81:0/432018976 learned_addr learned my addr 172.21.15.81:0/432018976 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:45:02.022 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.023+0000 7f9b877fe640 1 -- 172.21.15.81:0/432018976 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983485135 0 0) 0x7f9b90142520 con 0x7f9b901430c0 2024-03-25T19:45:02.022 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.023+0000 7f9b877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:02.022 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.023+0000 7f9b877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:02.022 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.023+0000 7f9b877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:02.022 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.023+0000 7f9b877fe640 10 monclient(hunting): my global_id is 4362 2024-03-25T19:45:02.022 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.023+0000 7f9b877fe640 1 -- 172.21.15.81:0/432018976 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b78003520 con 0x7f9b901430c0 2024-03-25T19:45:02.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.040+0000 7f9b877fe640 1 -- 172.21.15.81:0/432018976 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379852432 0 0) 0x7f9b900b2a30 con 0x7f9b9013f710 2024-03-25T19:45:02.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.040+0000 7f9b877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:02.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.040+0000 7f9b877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:02.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.040+0000 7f9b877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:02.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.040+0000 7f9b877fe640 10 monclient(hunting): my global_id is 4346 2024-03-25T19:45:02.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.040+0000 7f9b877fe640 1 -- 172.21.15.81:0/432018976 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b90142520 con 0x7f9b9013f710 2024-03-25T19:45:02.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b877fe640 1 -- 172.21.15.81:0/432018976 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2943072417 0 0) 0x7f9b78003520 con 0x7f9b901430c0 2024-03-25T19:45:02.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b877fe640 1 -- 172.21.15.81:0/432018976 >> v1:172.21.15.81:6790/0 conn(0x7f9b9013f710 legacy=0x7f9b901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b877fe640 1 -- 172.21.15.81:0/432018976 >> v1:172.21.15.98:6789/0 conn(0x7f9b900a98f0 legacy=0x7f9b900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b877fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:02.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b877fe640 1 monclient: found mon.noname-a 2024-03-25T19:45:02.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:02.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b877fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:45:02.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b877fe640 1 -- 172.21.15.81:0/432018976 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b900b44c0 con 0x7f9b901430c0 2024-03-25T19:45:02.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b877fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:02.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b877fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:02.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:02.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b9715d640 5 monclient: authenticate success, global_id 4362 2024-03-25T19:45:02.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b877fe640 1 -- 172.21.15.81:0/432018976 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f9b88001a10 con 0x7f9b901430c0 2024-03-25T19:45:02.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b9715d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:02.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:02.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:02.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b877fe640 10 monclient: dump: 2024-03-25T19:45:02.051 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:02.051 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:02.051 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:02.051 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:02.051 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:02.051 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:02.051 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:02.051 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:02.051 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:02.051 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:02.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.050+0000 7f9b9715d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:02.063 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.063+0000 7f9b877fe640 1 -- 172.21.15.81:0/432018976 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b88001b70 con 0x7f9b901430c0 2024-03-25T19:45:02.063 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.063+0000 7f9b877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:02.063 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.063+0000 7f9b877fe640 1 -- 172.21.15.81:0/432018976 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f9b88001f80 con 0x7f9b901430c0 2024-03-25T19:45:02.063 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.063+0000 7f9b877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:02.063 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.063+0000 7f9b877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:02.063 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.063+0000 7f9b877fe640 10 monclient: dump: 2024-03-25T19:45:02.063 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:02.063 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:02.063 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:02.063 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:02.064 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:02.064 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:02.064 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:02.064 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:02.064 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:02.064 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:02.064 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.063+0000 7f9b9715d640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:02.064 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.063+0000 7f9b9715d640 4 set_mon_vals no callback set 2024-03-25T19:45:02.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.064+0000 7f9b9715d640 10 monclient: shutdown 2024-03-25T19:45:02.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.064+0000 7f9b9715d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:02.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.064+0000 7f9b9715d640 1 -- 172.21.15.81:0/432018976 >> v1:172.21.15.81:6789/0 conn(0x7f9b901430c0 legacy=0x7f9b9014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.064+0000 7f9b9715d640 1 -- 172.21.15.81:0/432018976 shutdown_connections 2024-03-25T19:45:02.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.064+0000 7f9b9715d640 1 -- 172.21.15.81:0/432018976 >> 172.21.15.81:0/432018976 conn(0x7f9b900b7ea0 msgr2=0x7f9b900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:02.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.064+0000 7f9b9715d640 1 -- 172.21.15.81:0/432018976 shutdown_connections 2024-03-25T19:45:02.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.064+0000 7f9b9715d640 1 -- 172.21.15.81:0/432018976 wait complete. 2024-03-25T19:45:02.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.065+0000 7f9b9715d640 10 monclient: build_initial_monmap 2024-03-25T19:45:02.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.065+0000 7f9b9715d640 10 monclient: monmap: 2024-03-25T19:45:02.066 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:45:02.066 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:02.066 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:45:02.066 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:45:02.067 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:02.067 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:45:02.067 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:45:02.067 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:45:02.067 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:45:02.067 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:02.067 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.066+0000 7f9b9715d640 1 librados: starting msgr at 2024-03-25T19:45:02.067 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.066+0000 7f9b9715d640 1 librados: starting objecter 2024-03-25T19:45:02.067 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.066+0000 7f9b9715d640 1 Processor -- start 2024-03-25T19:45:02.067 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.066+0000 7f9b9715d640 1 -- start start 2024-03-25T19:45:02.067 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.066+0000 7f9b9715d640 1 librados: setting wanted keys 2024-03-25T19:45:02.067 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.066+0000 7f9b9715d640 1 librados: calling monclient init 2024-03-25T19:45:02.067 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.066+0000 7f9b9715d640 10 monclient: init 2024-03-25T19:45:02.068 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.067+0000 7f9b9715d640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:02.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.067+0000 7f9b9715d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:02.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.067+0000 7f9b9715d640 10 monclient(hunting): picked mon.noname-c con 0x7f9b900a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:45:02.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.067+0000 7f9b9715d640 10 monclient(hunting): picked mon.noname-b con 0x7f9b9013f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:45:02.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.067+0000 7f9b9715d640 10 monclient(hunting): picked mon.noname-a con 0x7f9b901430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:45:02.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.067+0000 7f9b9715d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b9013b410 con 0x7f9b901430c0 2024-03-25T19:45:02.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.067+0000 7f9b9715d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b9013b010 con 0x7f9b9013f710 2024-03-25T19:45:02.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.067+0000 7f9b9715d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b90135230 con 0x7f9b900a98f0 2024-03-25T19:45:02.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.067+0000 7f9b9715d640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:02.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.067+0000 7f9b9715d640 10 monclient(hunting): authenticate will time out at 4137.149308s 2024-03-25T19:45:02.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.074+0000 7f9b87fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9b9013f710 0x7f9b901dabe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:33354/0 (socket says 172.21.15.81:33354) 2024-03-25T19:45:02.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.074+0000 7f9b87fff640 1 -- 172.21.15.81:0/2096746815 learned_addr learned my addr 172.21.15.81:0/2096746815 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:45:02.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.075+0000 7f9b857fa640 10 client.?.objecter ms_handle_connect 0x7f9b9013f710 2024-03-25T19:45:02.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.075+0000 7f9b857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:02.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.078+0000 7f9b857fa640 10 client.?.objecter ms_handle_connect 0x7f9b901430c0 2024-03-25T19:45:02.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.078+0000 7f9b857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:02.079 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.080+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353316450 0 0) 0x7f9b9013b010 con 0x7f9b9013f710 2024-03-25T19:45:02.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.080+0000 7f9b857fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:02.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.080+0000 7f9b857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:02.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.080+0000 7f9b857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:02.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.080+0000 7f9b857fa640 10 monclient(hunting): my global_id is 4363 2024-03-25T19:45:02.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.080+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b7c004eb0 con 0x7f9b9013f710 2024-03-25T19:45:02.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.082+0000 7f9b857fa640 10 client.?.objecter ms_handle_connect 0x7f9b900a98f0 2024-03-25T19:45:02.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.082+0000 7f9b857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:02.085 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.086+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2395907806 0 0) 0x7f9b7c004eb0 con 0x7f9b9013f710 2024-03-25T19:45:02.085 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.086+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b9013b010 con 0x7f9b9013f710 2024-03-25T19:45:02.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.087+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825163366 0 0) 0x7f9b9013b410 con 0x7f9b901430c0 2024-03-25T19:45:02.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.087+0000 7f9b857fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:02.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.087+0000 7f9b857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:02.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.087+0000 7f9b857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:02.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.087+0000 7f9b857fa640 10 monclient(hunting): my global_id is 4365 2024-03-25T19:45:02.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.087+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b7c004eb0 con 0x7f9b901430c0 2024-03-25T19:45:02.087 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.088+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f9b80001af0 con 0x7f9b9013f710 2024-03-25T19:45:02.087 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.088+0000 7f9b857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:02.087 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.088+0000 7f9b857fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:02.088 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.088+0000 7f9b857fa640 10 monclient(hunting): dump: 2024-03-25T19:45:02.088 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:02.088 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:02.088 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:02.088 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:02.088 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:02.088 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:02.088 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:02.088 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:02.088 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:02.088 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:02.088 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.088+0000 7f9b857fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:02.092 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.093+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1616177415 0 0) 0x7f9b7c004eb0 con 0x7f9b901430c0 2024-03-25T19:45:02.092 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.093+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b9013b410 con 0x7f9b901430c0 2024-03-25T19:45:02.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 220452067 0 0) 0x7f9b9013b010 con 0x7f9b9013f710 2024-03-25T19:45:02.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 >> v1:172.21.15.81:6790/0 conn(0x7f9b900a98f0 legacy=0x7f9b90130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 >> v1:172.21.15.81:6789/0 conn(0x7f9b901430c0 legacy=0x7f9b901db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b857fa640 10 monclient: _finish_hunting 0 2024-03-25T19:45:02.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b857fa640 1 monclient: found mon.b 2024-03-25T19:45:02.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:02.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b857fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:02.098 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b901353c0 con 0x7f9b9013f710 2024-03-25T19:45:02.098 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b857fa640 10 monclient: _finish_auth 0 2024-03-25T19:45:02.098 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b857fa640 10 monclient: _check_auth_tickets 2024-03-25T19:45:02.098 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:02.098 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b9715d640 5 monclient: authenticate success, global_id 4363 2024-03-25T19:45:02.098 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b9715d640 10 monclient: _renew_subs 2024-03-25T19:45:02.098 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b9715d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:02.098 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b9715d640 1 -- 172.21.15.81:0/2096746815 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b900b78e0 con 0x7f9b9013f710 2024-03-25T19:45:02.098 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b9715d640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:02.098 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b9715d640 10 monclient: _renew_subs 2024-03-25T19:45:02.098 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b9715d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:02.098 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b9715d640 1 -- 172.21.15.81:0/2096746815 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b901317d0 con 0x7f9b9013f710 2024-03-25T19:45:02.098 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.097+0000 7f9b9715d640 1 librados: init done 2024-03-25T19:45:02.099 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.098+0000 7f9b9715d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:02.099 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.098+0000 7f9b9715d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:02.099 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.098+0000 7f9b9715d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:02.099 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.098+0000 7f9b9715d640 1 -- 172.21.15.81:0/2096746815 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b900b7c80 con 0x7f9b9013f710 2024-03-25T19:45:02.379 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.380+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b80001c50 con 0x7f9b9013f710 2024-03-25T19:45:02.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.380+0000 7f9b857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:02.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.380+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f9b80003ae0 con 0x7f9b9013f710 2024-03-25T19:45:02.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.380+0000 7f9b857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:02.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.380+0000 7f9b857fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:02.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.380+0000 7f9b857fa640 10 monclient: dump: 2024-03-25T19:45:02.380 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:02.380 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:02.380 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:02.380 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:02.380 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:02.381 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:02.381 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:02.381 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:02.381 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:02.381 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:02.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.380+0000 7f9b86ffd640 4 set_mon_vals no callback set 2024-03-25T19:45:02.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.381+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f9b80003d00 con 0x7f9b9013f710 2024-03-25T19:45:02.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.381+0000 7f9b94ef9640 10 monclient: get_auth_request con 0x7f9b7c06df30 auth_method 0 2024-03-25T19:45:02.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.381+0000 7f9b857fa640 10 client.4363.objecter ms_handle_connect 0x7f9b7c06df30 2024-03-25T19:45:02.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.383+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f9b80054a10 con 0x7f9b9013f710 2024-03-25T19:45:02.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.383+0000 7f9b857fa640 10 client.4363.objecter ms_dispatch 0x7f9b90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:02.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.383+0000 7f9b857fa640 3 client.4363.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:02.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.383+0000 7f9b857fa640 3 client.4363.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:02.384 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.383+0000 7f9b857fa640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-25T19:45:02.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.385+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b80050520 con 0x7f9b9013f710 2024-03-25T19:45:02.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.385+0000 7f9b857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:02.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.385+0000 7f9b857fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:02.616 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.616+0000 7f9b9715d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:02.616 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.616+0000 7f9b9715d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:02.616 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.616+0000 7f9b9715d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:02.617 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.616+0000 7f9b9715d640 1 -- 172.21.15.81:0/2096746815 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b9013b5c0 con 0x7f9b9013f710 2024-03-25T19:45:02.637 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.637+0000 7f9b857fa640 1 -- 172.21.15.81:0/2096746815 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f9b9013b5c0 con 0x7f9b9013f710 2024-03-25T19:45:02.637 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.637+0000 7f9b857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:02.637 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.637+0000 7f9b857fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:02.637 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:45:02.637 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:02.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.640+0000 7f9b9715d640 10 librados: watch_flush enter 2024-03-25T19:45:02.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.640+0000 7f9b9715d640 10 librados: watch_flush exit 2024-03-25T19:45:02.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.640+0000 7f9b9715d640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-25T19:45:02.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.640+0000 7f9b9715d640 10 client.4363.objecter successfully canceled tick 2024-03-25T19:45:02.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.640+0000 7f9b9715d640 1 -- 172.21.15.81:0/2096746815 >> v1:172.21.15.98:6800/3980829775 conn(0x7f9b7c06df30 legacy=0x7f9b7c0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.640+0000 7f9b9715d640 10 monclient: shutdown 2024-03-25T19:45:02.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.640+0000 7f9b9715d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:02.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.640+0000 7f9b9715d640 1 -- 172.21.15.81:0/2096746815 >> v1:172.21.15.98:6789/0 conn(0x7f9b9013f710 legacy=0x7f9b901dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.640+0000 7f9b9715d640 1 -- 172.21.15.81:0/2096746815 shutdown_connections 2024-03-25T19:45:02.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.640+0000 7f9b9715d640 1 -- 172.21.15.81:0/2096746815 >> 172.21.15.81:0/2096746815 conn(0x7f9b900b7ea0 msgr2=0x7f9b90074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:02.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.640+0000 7f9b9715d640 1 -- 172.21.15.81:0/2096746815 shutdown_connections 2024-03-25T19:45:02.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.640+0000 7f9b9715d640 1 -- 172.21.15.81:0/2096746815 wait complete. 2024-03-25T19:45:02.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:02.640+0000 7f9b9715d640 1 librados: shutdown 2024-03-25T19:45:02.652 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:09.458 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:09.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.565+0000 7fd81ef32640 10 monclient: get_monmap_and_config 2024-03-25T19:45:09.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.565+0000 7fd81ef32640 10 monclient: build_initial_monmap 2024-03-25T19:45:09.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.565+0000 7fd81ef32640 10 monclient: monmap: 2024-03-25T19:45:09.566 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:45:09.566 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:09.567 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:45:09.567635+0000 2024-03-25T19:45:09.567 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:45:09.567635+0000 2024-03-25T19:45:09.567 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:09.567 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:09.567 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:45:09.567 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:45:09.567 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:45:09.567 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.566+0000 7fd81ef32640 1 Processor -- start 2024-03-25T19:45:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.566+0000 7fd81ef32640 1 -- start start 2024-03-25T19:45:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.566+0000 7fd81ef32640 10 monclient: init 2024-03-25T19:45:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.567+0000 7fd81ef32640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.567+0000 7fd81ef32640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.567+0000 7fd81ef32640 10 monclient(hunting): picked mon.noname-b con 0x7fd8180a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:45:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.567+0000 7fd81ef32640 10 monclient(hunting): picked mon.noname-a con 0x7fd81813f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:45:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.567+0000 7fd81ef32640 10 monclient(hunting): picked mon.noname-c con 0x7fd8181430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:45:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.567+0000 7fd81ef32640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd818142520 con 0x7fd81813f710 2024-03-25T19:45:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.567+0000 7fd81ef32640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8180b28a0 con 0x7fd8181430c0 2024-03-25T19:45:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.567+0000 7fd81ef32640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8180b2a30 con 0x7fd8180a98f0 2024-03-25T19:45:09.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.567+0000 7fd81ef32640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:09.570 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.567+0000 7fd81ef32640 10 monclient(hunting): authenticate will time out at 4144.650692s 2024-03-25T19:45:09.576 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.575+0000 7fd81ccce640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fd8180a98f0 0x7fd8180a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:46802/0 (socket says 172.21.15.81:46802) 2024-03-25T19:45:09.576 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.575+0000 7fd81ccce640 1 -- 172.21.15.81:0/173546657 learned_addr learned my addr 172.21.15.81:0/173546657 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:45:09.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.591+0000 7fd8177fe640 1 -- 172.21.15.81:0/173546657 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989733473 0 0) 0x7fd8180b2a30 con 0x7fd8180a98f0 2024-03-25T19:45:09.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.591+0000 7fd8177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:09.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.591+0000 7fd8177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:09.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.591+0000 7fd8177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:09.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.591+0000 7fd8177fe640 10 monclient(hunting): my global_id is 4352 2024-03-25T19:45:09.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.591+0000 7fd8177fe640 1 -- 172.21.15.81:0/173546657 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd808003520 con 0x7fd8180a98f0 2024-03-25T19:45:09.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.597+0000 7fd8177fe640 1 -- 172.21.15.81:0/173546657 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003970604 0 0) 0x7fd818142520 con 0x7fd81813f710 2024-03-25T19:45:09.598 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.597+0000 7fd8177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:09.598 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.597+0000 7fd8177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:09.598 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.597+0000 7fd8177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:09.598 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.597+0000 7fd8177fe640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:45:09.598 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.597+0000 7fd8177fe640 1 -- 172.21.15.81:0/173546657 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8180b2a30 con 0x7fd81813f710 2024-03-25T19:45:09.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.598+0000 7fd8177fe640 1 -- 172.21.15.81:0/173546657 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986617192 0 0) 0x7fd8180b28a0 con 0x7fd8181430c0 2024-03-25T19:45:09.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.598+0000 7fd8177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:09.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.598+0000 7fd8177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:09.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.598+0000 7fd8177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:09.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.598+0000 7fd8177fe640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:45:09.599 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.598+0000 7fd8177fe640 1 -- 172.21.15.81:0/173546657 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd818142520 con 0x7fd8181430c0 2024-03-25T19:45:09.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd8177fe640 1 -- 172.21.15.81:0/173546657 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1518645268 0 0) 0x7fd808003520 con 0x7fd8180a98f0 2024-03-25T19:45:09.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd8177fe640 1 -- 172.21.15.81:0/173546657 >> v1:172.21.15.98:6789/0 conn(0x7fd8181430c0 legacy=0x7fd81814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:09.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd8177fe640 1 -- 172.21.15.81:0/173546657 >> v1:172.21.15.81:6789/0 conn(0x7fd81813f710 legacy=0x7fd8181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:09.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd8177fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:09.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd8177fe640 1 monclient: found mon.noname-b 2024-03-25T19:45:09.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd8177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:09.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd8177fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:45:09.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd8177fe640 1 -- 172.21.15.81:0/173546657 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8180b44c0 con 0x7fd8180a98f0 2024-03-25T19:45:09.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd8177fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:09.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd8177fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:09.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd8177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:09.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd8177fe640 1 -- 172.21.15.81:0/173546657 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd80c001690 con 0x7fd8180a98f0 2024-03-25T19:45:09.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd81ef32640 5 monclient: authenticate success, global_id 4352 2024-03-25T19:45:09.612 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd81ef32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:09.612 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd8177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:09.612 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd8177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:09.612 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd8177fe640 10 monclient: dump: 2024-03-25T19:45:09.612 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:09.612 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:09.612 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:09.612 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:09.612 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:09.612 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:09.612 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:09.612 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:09.612 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:09.612 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:09.612 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.610+0000 7fd81ef32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:09.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.620+0000 7fd8177fe640 1 -- 172.21.15.81:0/173546657 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd80c0017f0 con 0x7fd8180a98f0 2024-03-25T19:45:09.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.620+0000 7fd8177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:09.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.620+0000 7fd8177fe640 1 -- 172.21.15.81:0/173546657 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd80c001c00 con 0x7fd8180a98f0 2024-03-25T19:45:09.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.620+0000 7fd8177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:09.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.620+0000 7fd8177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:09.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.620+0000 7fd8177fe640 10 monclient: dump: 2024-03-25T19:45:09.621 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:09.622 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:09.622 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:09.622 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:09.622 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:09.622 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:09.622 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:09.622 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:09.622 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:09.622 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:09.622 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.620+0000 7fd81ef32640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:09.622 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.620+0000 7fd81ef32640 4 set_mon_vals no callback set 2024-03-25T19:45:09.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.621+0000 7fd81ef32640 10 monclient: shutdown 2024-03-25T19:45:09.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.621+0000 7fd81ef32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:09.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.621+0000 7fd81ef32640 1 -- 172.21.15.81:0/173546657 >> v1:172.21.15.81:6790/0 conn(0x7fd8180a98f0 legacy=0x7fd8180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:09.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.621+0000 7fd81ef32640 1 -- 172.21.15.81:0/173546657 shutdown_connections 2024-03-25T19:45:09.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.621+0000 7fd81ef32640 1 -- 172.21.15.81:0/173546657 >> 172.21.15.81:0/173546657 conn(0x7fd8180b7ea0 msgr2=0x7fd8180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:09.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.621+0000 7fd81ef32640 1 -- 172.21.15.81:0/173546657 shutdown_connections 2024-03-25T19:45:09.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.621+0000 7fd81ef32640 1 -- 172.21.15.81:0/173546657 wait complete. 2024-03-25T19:45:09.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.622+0000 7fd81ef32640 10 monclient: build_initial_monmap 2024-03-25T19:45:09.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.622+0000 7fd81ef32640 10 monclient: monmap: 2024-03-25T19:45:09.625 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:45:09.625 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:09.625 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:45:09.626 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:45:09.626 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:09.626 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:45:09.626 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:45:09.626 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:45:09.626 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:45:09.626 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:09.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.622+0000 7fd81ef32640 1 librados: starting msgr at 2024-03-25T19:45:09.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.622+0000 7fd81ef32640 1 librados: starting objecter 2024-03-25T19:45:09.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.622+0000 7fd81ef32640 1 Processor -- start 2024-03-25T19:45:09.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.622+0000 7fd81ef32640 1 -- start start 2024-03-25T19:45:09.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.622+0000 7fd81ef32640 1 librados: setting wanted keys 2024-03-25T19:45:09.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.622+0000 7fd81ef32640 1 librados: calling monclient init 2024-03-25T19:45:09.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.622+0000 7fd81ef32640 10 monclient: init 2024-03-25T19:45:09.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.623+0000 7fd81ef32640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:09.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.623+0000 7fd81ef32640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:09.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.623+0000 7fd81ef32640 10 monclient(hunting): picked mon.noname-b con 0x7fd8180a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:45:09.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.623+0000 7fd81ef32640 10 monclient(hunting): picked mon.noname-c con 0x7fd81813f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:45:09.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.623+0000 7fd81ef32640 10 monclient(hunting): picked mon.noname-a con 0x7fd8181430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:45:09.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.623+0000 7fd81ef32640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8181db7d0 con 0x7fd8181430c0 2024-03-25T19:45:09.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.623+0000 7fd81ef32640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8180b1420 con 0x7fd8180a98f0 2024-03-25T19:45:09.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.623+0000 7fd81ef32640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8180b15b0 con 0x7fd81813f710 2024-03-25T19:45:09.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.623+0000 7fd81ef32640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:09.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.623+0000 7fd81ef32640 10 monclient(hunting): authenticate will time out at 4144.706535s 2024-03-25T19:45:09.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.633+0000 7fd81d4cf640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fd8181430c0 0x7fd8181d4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:50890/0 (socket says 172.21.15.81:50890) 2024-03-25T19:45:09.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.633+0000 7fd81d4cf640 1 -- 172.21.15.81:0/2332238572 learned_addr learned my addr 172.21.15.81:0/2332238572 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:45:09.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.636+0000 7fd8157fa640 10 client.?.objecter ms_handle_connect 0x7fd8181430c0 2024-03-25T19:45:09.637 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.636+0000 7fd8157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:09.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.642+0000 7fd8157fa640 10 client.?.objecter ms_handle_connect 0x7fd81813f710 2024-03-25T19:45:09.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.642+0000 7fd8157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:09.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.643+0000 7fd8157fa640 1 -- 172.21.15.81:0/2332238572 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1931059794 0 0) 0x7fd8181db7d0 con 0x7fd8181430c0 2024-03-25T19:45:09.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.643+0000 7fd8157fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:09.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.643+0000 7fd8157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:09.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.643+0000 7fd8157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:09.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.643+0000 7fd8157fa640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:45:09.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.643+0000 7fd8157fa640 1 -- 172.21.15.81:0/2332238572 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7fc004eb0 con 0x7fd8181430c0 2024-03-25T19:45:09.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.645+0000 7fd8157fa640 10 client.?.objecter ms_handle_connect 0x7fd8180a98f0 2024-03-25T19:45:09.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.645+0000 7fd8157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:09.655 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.655+0000 7fd8157fa640 1 -- 172.21.15.81:0/2332238572 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1452679249 0 0) 0x7fd7fc004eb0 con 0x7fd8181430c0 2024-03-25T19:45:09.655 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.655+0000 7fd8157fa640 1 -- 172.21.15.81:0/2332238572 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8181db7d0 con 0x7fd8181430c0 2024-03-25T19:45:09.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.655+0000 7fd8157fa640 1 -- 172.21.15.81:0/2332238572 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd810001c10 con 0x7fd8181430c0 2024-03-25T19:45:09.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.655+0000 7fd8157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:09.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.655+0000 7fd8157fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:09.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.655+0000 7fd8157fa640 10 monclient(hunting): dump: 2024-03-25T19:45:09.656 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:09.656 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:09.656 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:09.656 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:09.656 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:09.656 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:09.656 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:09.656 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:09.656 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:09.656 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:09.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.655+0000 7fd8157fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:09.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.656+0000 7fd8157fa640 1 -- 172.21.15.81:0/2332238572 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899061802 0 0) 0x7fd8180b15b0 con 0x7fd81813f710 2024-03-25T19:45:09.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.656+0000 7fd8157fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:09.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.656+0000 7fd8157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:09.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.656+0000 7fd8157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:09.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.656+0000 7fd8157fa640 10 monclient(hunting): my global_id is 4355 2024-03-25T19:45:09.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.656+0000 7fd8157fa640 1 -- 172.21.15.81:0/2332238572 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7fc004eb0 con 0x7fd81813f710 2024-03-25T19:45:09.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd8157fa640 1 -- 172.21.15.81:0/2332238572 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 792907858 0 0) 0x7fd8181db7d0 con 0x7fd8181430c0 2024-03-25T19:45:09.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd8157fa640 1 -- 172.21.15.81:0/2332238572 >> v1:172.21.15.81:6790/0 conn(0x7fd81813f710 legacy=0x7fd8180b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:09.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd8157fa640 1 -- 172.21.15.81:0/2332238572 >> v1:172.21.15.98:6789/0 conn(0x7fd8180a98f0 legacy=0x7fd8181daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:09.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd8157fa640 10 monclient: _finish_hunting 0 2024-03-25T19:45:09.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd8157fa640 1 monclient: found mon.a 2024-03-25T19:45:09.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd8157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:09.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd8157fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:45:09.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd8157fa640 1 -- 172.21.15.81:0/2332238572 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8180b1740 con 0x7fd8181430c0 2024-03-25T19:45:09.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd8157fa640 10 monclient: _finish_auth 0 2024-03-25T19:45:09.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd8157fa640 10 monclient: _check_auth_tickets 2024-03-25T19:45:09.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd8157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:09.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd81ef32640 5 monclient: authenticate success, global_id 4371 2024-03-25T19:45:09.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd81ef32640 10 monclient: _renew_subs 2024-03-25T19:45:09.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd81ef32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:45:09.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd81ef32640 1 -- 172.21.15.81:0/2332238572 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8181db1a0 con 0x7fd8181430c0 2024-03-25T19:45:09.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd81ef32640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:09.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd81ef32640 10 monclient: _renew_subs 2024-03-25T19:45:09.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd81ef32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:45:09.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd81ef32640 1 -- 172.21.15.81:0/2332238572 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8180b08f0 con 0x7fd8181430c0 2024-03-25T19:45:09.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.661+0000 7fd81ef32640 1 librados: init done 2024-03-25T19:45:09.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.662+0000 7fd81ef32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:09.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.662+0000 7fd81ef32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:09.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.662+0000 7fd81ef32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:45:09.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.662+0000 7fd81ef32640 1 -- 172.21.15.81:0/2332238572 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8180627b0 con 0x7fd8181430c0 2024-03-25T19:45:09.962 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.961+0000 7fd8157fa640 1 -- 172.21.15.81:0/2332238572 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd810003990 con 0x7fd8181430c0 2024-03-25T19:45:09.962 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.961+0000 7fd8157fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:09.962 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.961+0000 7fd8157fa640 1 -- 172.21.15.81:0/2332238572 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd810003d20 con 0x7fd8181430c0 2024-03-25T19:45:09.962 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.961+0000 7fd8157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:09.962 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.961+0000 7fd8157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:09.962 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.961+0000 7fd8157fa640 10 monclient: dump: 2024-03-25T19:45:09.962 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:09.962 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:09.962 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:09.963 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:09.963 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:09.963 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:09.963 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:09.963 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:09.963 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:09.963 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:09.963 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.961+0000 7fd816ffd640 4 set_mon_vals no callback set 2024-03-25T19:45:09.964 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.962+0000 7fd8157fa640 1 -- 172.21.15.81:0/2332238572 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fd810003f40 con 0x7fd8181430c0 2024-03-25T19:45:09.964 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.962+0000 7fd81ccce640 10 monclient: get_auth_request con 0x7fd7fc06dd90 auth_method 0 2024-03-25T19:45:09.964 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.963+0000 7fd8157fa640 10 client.4371.objecter ms_handle_connect 0x7fd7fc06dd90 2024-03-25T19:45:09.968 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.968+0000 7fd8157fa640 1 -- 172.21.15.81:0/2332238572 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fd810054a90 con 0x7fd8181430c0 2024-03-25T19:45:09.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.968+0000 7fd8157fa640 10 client.4371.objecter ms_dispatch 0x7fd818000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:09.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.968+0000 7fd8157fa640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:09.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.968+0000 7fd8157fa640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:09.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.968+0000 7fd8157fa640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-25T19:45:09.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.968+0000 7fd8157fa640 1 -- 172.21.15.81:0/2332238572 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd8100505a0 con 0x7fd8181430c0 2024-03-25T19:45:09.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.968+0000 7fd8157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:09.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:09.968+0000 7fd8157fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:10.218 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.218+0000 7fd81ef32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:10.219 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.218+0000 7fd81ef32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:10.219 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.218+0000 7fd81ef32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:45:10.219 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.218+0000 7fd81ef32640 1 -- 172.21.15.81:0/2332238572 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8180abb10 con 0x7fd8181430c0 2024-03-25T19:45:10.245 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.245+0000 7fd8157fa640 1 -- 172.21.15.81:0/2332238572 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fd8100524d0 con 0x7fd8181430c0 2024-03-25T19:45:10.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.245+0000 7fd8157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:10.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.245+0000 7fd8157fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:10.246 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:45:10.246 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:10.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.247+0000 7fd81ef32640 10 librados: watch_flush enter 2024-03-25T19:45:10.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.247+0000 7fd81ef32640 10 librados: watch_flush exit 2024-03-25T19:45:10.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.247+0000 7fd81ef32640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-25T19:45:10.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.247+0000 7fd81ef32640 10 client.4371.objecter successfully canceled tick 2024-03-25T19:45:10.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.247+0000 7fd81ef32640 1 -- 172.21.15.81:0/2332238572 >> v1:172.21.15.98:6800/3980829775 conn(0x7fd7fc06dd90 legacy=0x7fd7fc070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.248+0000 7fd81ef32640 10 monclient: shutdown 2024-03-25T19:45:10.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.248+0000 7fd81ef32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:10.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.248+0000 7fd81ef32640 1 -- 172.21.15.81:0/2332238572 >> v1:172.21.15.81:6789/0 conn(0x7fd8181430c0 legacy=0x7fd8181d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.248+0000 7fd81ef32640 1 -- 172.21.15.81:0/2332238572 shutdown_connections 2024-03-25T19:45:10.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.248+0000 7fd81ef32640 1 -- 172.21.15.81:0/2332238572 >> 172.21.15.81:0/2332238572 conn(0x7fd8180b7ea0 msgr2=0x7fd81813c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:10.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.248+0000 7fd81ef32640 1 -- 172.21.15.81:0/2332238572 shutdown_connections 2024-03-25T19:45:10.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.248+0000 7fd81ef32640 1 -- 172.21.15.81:0/2332238572 wait complete. 2024-03-25T19:45:10.250 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:10.248+0000 7fd81ef32640 1 librados: shutdown 2024-03-25T19:45:10.260 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:17.067 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:17.084 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:03:39.726 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:17.132 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:03:39.774 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:17.157 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:03:39.799 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:17.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.174+0000 7efc1abb5640 10 monclient: get_monmap_and_config 2024-03-25T19:45:17.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.174+0000 7efc1abb5640 10 monclient: build_initial_monmap 2024-03-25T19:45:17.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.174+0000 7efc1abb5640 10 monclient: monmap: 2024-03-25T19:45:17.174 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:45:17.174 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:17.174 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:45:17.175467+0000 2024-03-25T19:45:17.174 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:45:17.175467+0000 2024-03-25T19:45:17.174 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:17.174 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:17.174 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:45:17.174 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:45:17.174 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:45:17.175 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:17.176 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.175+0000 7efc1abb5640 1 Processor -- start 2024-03-25T19:45:17.176 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.175+0000 7efc1abb5640 1 -- start start 2024-03-25T19:45:17.176 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.175+0000 7efc1abb5640 10 monclient: init 2024-03-25T19:45:17.176 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.175+0000 7efc1abb5640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:17.176 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.175+0000 7efc1abb5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:17.176 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.175+0000 7efc1abb5640 10 monclient(hunting): picked mon.noname-b con 0x7efc140b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:45:17.176 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.175+0000 7efc1abb5640 10 monclient(hunting): picked mon.noname-c con 0x7efc140b1730 addr v1:172.21.15.98:6789/0 2024-03-25T19:45:17.176 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.175+0000 7efc1abb5640 10 monclient(hunting): picked mon.noname-a con 0x7efc1414be90 addr v1:172.21.15.81:6789/0 2024-03-25T19:45:17.176 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.175+0000 7efc1abb5640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc140af940 con 0x7efc1414be90 2024-03-25T19:45:17.176 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.176+0000 7efc1abb5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc140b0a70 con 0x7efc140b1730 2024-03-25T19:45:17.176 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.176+0000 7efc1abb5640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc140b0d50 con 0x7efc140b4d60 2024-03-25T19:45:17.176 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.176+0000 7efc1abb5640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:17.176 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.176+0000 7efc1abb5640 10 monclient(hunting): authenticate will time out at 4152.257858s 2024-03-25T19:45:17.188 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:39.823 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:17.195 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:39.830 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:17.200 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:03:39.842 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:17.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.201+0000 7efc19152640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7efc1414be90 0x7efc1414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:55798/0 (socket says 172.21.15.81:55798) 2024-03-25T19:45:17.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.201+0000 7efc19152640 1 -- 172.21.15.81:0/2632567989 learned_addr learned my addr 172.21.15.81:0/2632567989 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:45:17.206 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:03:39.848 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:17.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.224+0000 7efc137fe640 1 -- 172.21.15.81:0/2632567989 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675040404 0 0) 0x7efc140af940 con 0x7efc1414be90 2024-03-25T19:45:17.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.224+0000 7efc137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:17.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.224+0000 7efc137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:17.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.224+0000 7efc137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:17.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.224+0000 7efc137fe640 10 monclient(hunting): my global_id is 4374 2024-03-25T19:45:17.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.224+0000 7efc137fe640 1 -- 172.21.15.81:0/2632567989 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc04003520 con 0x7efc1414be90 2024-03-25T19:45:17.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.232+0000 7efc137fe640 1 -- 172.21.15.81:0/2632567989 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158618751 0 0) 0x7efc140b0a70 con 0x7efc140b1730 2024-03-25T19:45:17.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.232+0000 7efc137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:17.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.232+0000 7efc137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:17.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.232+0000 7efc137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:17.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.232+0000 7efc137fe640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:45:17.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.232+0000 7efc137fe640 1 -- 172.21.15.81:0/2632567989 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc140af940 con 0x7efc140b1730 2024-03-25T19:45:17.236 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.235+0000 7efc137fe640 1 -- 172.21.15.81:0/2632567989 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 915265959 0 0) 0x7efc04003520 con 0x7efc1414be90 2024-03-25T19:45:17.236 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.235+0000 7efc137fe640 1 -- 172.21.15.81:0/2632567989 >> v1:172.21.15.81:6790/0 conn(0x7efc140b4d60 legacy=0x7efc1413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.236 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.235+0000 7efc137fe640 1 -- 172.21.15.81:0/2632567989 >> v1:172.21.15.98:6789/0 conn(0x7efc140b1730 legacy=0x7efc140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.236 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.235+0000 7efc137fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:17.236 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.235+0000 7efc137fe640 1 monclient: found mon.noname-a 2024-03-25T19:45:17.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.235+0000 7efc137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:17.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.235+0000 7efc137fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:45:17.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.235+0000 7efc137fe640 1 -- 172.21.15.81:0/2632567989 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc140b1030 con 0x7efc1414be90 2024-03-25T19:45:17.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.235+0000 7efc137fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:17.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.235+0000 7efc137fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:17.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.235+0000 7efc137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:17.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.235+0000 7efc137fe640 1 -- 172.21.15.81:0/2632567989 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7efc0c001a10 con 0x7efc1414be90 2024-03-25T19:45:17.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.235+0000 7efc137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:17.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.236+0000 7efc137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:17.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.236+0000 7efc137fe640 10 monclient: dump: 2024-03-25T19:45:17.238 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:17.238 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:17.238 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:17.238 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:17.238 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:17.238 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:17.238 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:17.238 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:17.238 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:17.238 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:17.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.236+0000 7efc1abb5640 5 monclient: authenticate success, global_id 4374 2024-03-25T19:45:17.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.236+0000 7efc1abb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:17.239 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:39.871 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:17.249 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:03:39.890 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:17.273 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.274+0000 7efc137fe640 1 -- 172.21.15.81:0/2632567989 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc0c001b70 con 0x7efc1414be90 2024-03-25T19:45:17.274 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.274+0000 7efc137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:17.274 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.274+0000 7efc137fe640 1 -- 172.21.15.81:0/2632567989 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7efc0c001f80 con 0x7efc1414be90 2024-03-25T19:45:17.274 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.274+0000 7efc1abb5640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:17.274 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.274+0000 7efc1abb5640 4 set_mon_vals no callback set 2024-03-25T19:45:17.274 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.274+0000 7efc137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:17.274 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.274+0000 7efc137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:17.274 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.274+0000 7efc137fe640 10 monclient: dump: 2024-03-25T19:45:17.274 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:17.274 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:17.274 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:17.274 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:17.274 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:17.275 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:17.275 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:17.275 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:17.275 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:17.275 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:17.277 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.274+0000 7efc1abb5640 10 monclient: shutdown 2024-03-25T19:45:17.277 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.274+0000 7efc1abb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:17.277 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.274+0000 7efc1abb5640 1 -- 172.21.15.81:0/2632567989 >> v1:172.21.15.81:6789/0 conn(0x7efc1414be90 legacy=0x7efc1414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.277 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.275+0000 7efc1abb5640 1 -- 172.21.15.81:0/2632567989 shutdown_connections 2024-03-25T19:45:17.277 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.275+0000 7efc1abb5640 1 -- 172.21.15.81:0/2632567989 >> 172.21.15.81:0/2632567989 conn(0x7efc140617a0 msgr2=0x7efc14061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:17.277 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.275+0000 7efc1abb5640 1 -- 172.21.15.81:0/2632567989 shutdown_connections 2024-03-25T19:45:17.277 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.275+0000 7efc1abb5640 1 -- 172.21.15.81:0/2632567989 wait complete. 2024-03-25T19:45:17.277 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.275+0000 7efc1abb5640 10 monclient: build_initial_monmap 2024-03-25T19:45:17.277 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.275+0000 7efc1abb5640 10 monclient: monmap: 2024-03-25T19:45:17.277 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:45:17.278 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:17.278 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:45:17.278 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:45:17.278 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:17.278 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:45:17.278 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:45:17.278 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:45:17.278 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:45:17.278 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:17.278 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.275+0000 7efc1abb5640 1 librados: starting msgr at 2024-03-25T19:45:17.278 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.275+0000 7efc1abb5640 1 librados: starting objecter 2024-03-25T19:45:17.278 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.276+0000 7efc1abb5640 1 Processor -- start 2024-03-25T19:45:17.278 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.276+0000 7efc1abb5640 1 -- start start 2024-03-25T19:45:17.279 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.276+0000 7efc1abb5640 1 librados: setting wanted keys 2024-03-25T19:45:17.279 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.276+0000 7efc1abb5640 1 librados: calling monclient init 2024-03-25T19:45:17.279 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.276+0000 7efc1abb5640 10 monclient: init 2024-03-25T19:45:17.279 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.276+0000 7efc1abb5640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:17.279 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.276+0000 7efc1abb5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:17.279 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.276+0000 7efc1abb5640 10 monclient(hunting): picked mon.noname-c con 0x7efc140b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:45:17.279 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.276+0000 7efc1abb5640 10 monclient(hunting): picked mon.noname-a con 0x7efc140b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:45:17.279 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.276+0000 7efc1abb5640 10 monclient(hunting): picked mon.noname-b con 0x7efc1414be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:45:17.279 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.276+0000 7efc1abb5640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc141d86f0 con 0x7efc140b4d60 2024-03-25T19:45:17.279 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.276+0000 7efc1abb5640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1413b4f0 con 0x7efc1414be90 2024-03-25T19:45:17.279 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.276+0000 7efc1abb5640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc141d7eb0 con 0x7efc140b1730 2024-03-25T19:45:17.279 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.276+0000 7efc1abb5640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:17.279 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.276+0000 7efc1abb5640 10 monclient(hunting): authenticate will time out at 4152.358702s 2024-03-25T19:45:17.283 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.284+0000 7efc19152640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efc1414be90 0x7efc141e3000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:60606/0 (socket says 172.21.15.81:60606) 2024-03-25T19:45:17.283 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.284+0000 7efc19152640 1 -- 172.21.15.81:0/2565375073 learned_addr learned my addr 172.21.15.81:0/2565375073 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:45:17.286 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.287+0000 7efc117fa640 10 client.?.objecter ms_handle_connect 0x7efc1414be90 2024-03-25T19:45:17.286 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.287+0000 7efc117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:17.304 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.305+0000 7efc117fa640 10 client.?.objecter ms_handle_connect 0x7efc140b1730 2024-03-25T19:45:17.304 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.305+0000 7efc117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:17.316 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.316+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735714551 0 0) 0x7efc1413b4f0 con 0x7efc1414be90 2024-03-25T19:45:17.316 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.316+0000 7efc117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:17.316 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.316+0000 7efc117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:17.316 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.316+0000 7efc117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:17.316 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.316+0000 7efc117fa640 10 monclient(hunting): my global_id is 4375 2024-03-25T19:45:17.316 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.316+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf8004eb0 con 0x7efc1414be90 2024-03-25T19:45:17.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.321+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705850282 0 0) 0x7efc141d7eb0 con 0x7efc140b1730 2024-03-25T19:45:17.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.321+0000 7efc117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:17.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.321+0000 7efc117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:17.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.321+0000 7efc117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:17.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.321+0000 7efc117fa640 10 monclient(hunting): my global_id is 4361 2024-03-25T19:45:17.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.321+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc1413b4f0 con 0x7efc140b1730 2024-03-25T19:45:17.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.321+0000 7efc117fa640 10 client.?.objecter ms_handle_connect 0x7efc140b4d60 2024-03-25T19:45:17.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.321+0000 7efc117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:17.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.330+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283242730 0 0) 0x7efc140af940 con 0x7efc140b4d60 2024-03-25T19:45:17.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.330+0000 7efc117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:17.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.330+0000 7efc117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:17.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.330+0000 7efc117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:17.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.330+0000 7efc117fa640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:45:17.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.330+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc141d7eb0 con 0x7efc140b4d60 2024-03-25T19:45:17.352 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.352+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2776503316 0 0) 0x7efbf8004eb0 con 0x7efc1414be90 2024-03-25T19:45:17.352 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.352+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc140af940 con 0x7efc1414be90 2024-03-25T19:45:17.352 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.352+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7efc0c001f40 con 0x7efc1414be90 2024-03-25T19:45:17.352 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.352+0000 7efc117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:17.352 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.352+0000 7efc117fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:17.352 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.352+0000 7efc117fa640 10 monclient(hunting): dump: 2024-03-25T19:45:17.352 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:17.352 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:17.353 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:17.353 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:17.353 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:17.353 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:17.353 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:17.353 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:17.353 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:17.353 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:17.353 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.352+0000 7efc117fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:17.361 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3362161778 0 0) 0x7efc140af940 con 0x7efc1414be90 2024-03-25T19:45:17.361 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 >> v1:172.21.15.81:6790/0 conn(0x7efc140b1730 legacy=0x7efc141d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.361 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 >> v1:172.21.15.81:6789/0 conn(0x7efc140b4d60 legacy=0x7efc14136fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.361 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc117fa640 10 monclient: _finish_hunting 0 2024-03-25T19:45:17.361 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc117fa640 1 monclient: found mon.b 2024-03-25T19:45:17.361 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:17.361 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc117fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:17.361 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc141d8040 con 0x7efc1414be90 2024-03-25T19:45:17.361 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc117fa640 10 monclient: _finish_auth 0 2024-03-25T19:45:17.361 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc117fa640 10 monclient: _check_auth_tickets 2024-03-25T19:45:17.361 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:17.361 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc1abb5640 5 monclient: authenticate success, global_id 4375 2024-03-25T19:45:17.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc1abb5640 10 monclient: _renew_subs 2024-03-25T19:45:17.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc1abb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:17.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc1abb5640 1 -- 172.21.15.81:0/2565375073 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1413ba70 con 0x7efc1414be90 2024-03-25T19:45:17.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc1abb5640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:17.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc1abb5640 10 monclient: _renew_subs 2024-03-25T19:45:17.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc1abb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:17.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc1abb5640 1 -- 172.21.15.81:0/2565375073 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc141378d0 con 0x7efc1414be90 2024-03-25T19:45:17.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.361+0000 7efc1abb5640 1 librados: init done 2024-03-25T19:45:17.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.362+0000 7efc1abb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:17.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.362+0000 7efc1abb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:17.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.362+0000 7efc1abb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:17.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.362+0000 7efc1abb5640 1 -- 172.21.15.81:0/2565375073 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc140627b0 con 0x7efc1414be90 2024-03-25T19:45:17.653 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.653+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc0c004e10 con 0x7efc1414be90 2024-03-25T19:45:17.653 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.653+0000 7efc117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:17.653 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.653+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7efc0c005120 con 0x7efc1414be90 2024-03-25T19:45:17.653 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.653+0000 7efc117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:17.653 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.653+0000 7efc117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:17.653 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.653+0000 7efc117fa640 10 monclient: dump: 2024-03-25T19:45:17.653 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:17.653 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:17.653 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:17.654 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:17.654 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:17.654 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:17.654 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:17.654 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:17.654 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:17.654 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:17.654 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.653+0000 7efc12ffd640 4 set_mon_vals no callback set 2024-03-25T19:45:17.655 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.654+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7efc0c01b0f0 con 0x7efc1414be90 2024-03-25T19:45:17.655 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.654+0000 7efc18951640 10 monclient: get_auth_request con 0x7efbf806de00 auth_method 0 2024-03-25T19:45:17.655 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.655+0000 7efc117fa640 10 client.4375.objecter ms_handle_connect 0x7efbf806de00 2024-03-25T19:45:17.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.665+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7efc0c054b50 con 0x7efc1414be90 2024-03-25T19:45:17.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.665+0000 7efc117fa640 10 client.4375.objecter ms_dispatch 0x7efc14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:17.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.665+0000 7efc117fa640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:17.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.665+0000 7efc117fa640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:17.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.665+0000 7efc117fa640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-25T19:45:17.665 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.665+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc0c050660 con 0x7efc1414be90 2024-03-25T19:45:17.665 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.665+0000 7efc117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:17.665 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.665+0000 7efc117fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:17.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.912+0000 7efc1abb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:17.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.912+0000 7efc1abb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:17.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.912+0000 7efc1abb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:17.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.912+0000 7efc1abb5640 1 -- 172.21.15.81:0/2565375073 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc14069f40 con 0x7efc1414be90 2024-03-25T19:45:17.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.933+0000 7efc117fa640 1 -- 172.21.15.81:0/2565375073 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7efc0c052590 con 0x7efc1414be90 2024-03-25T19:45:17.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.933+0000 7efc117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:17.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.933+0000 7efc117fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:17.933 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:45:17.933 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:17.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.935+0000 7efc1abb5640 10 librados: watch_flush enter 2024-03-25T19:45:17.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.936+0000 7efc1abb5640 10 librados: watch_flush exit 2024-03-25T19:45:17.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.936+0000 7efc1abb5640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-25T19:45:17.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.936+0000 7efc1abb5640 10 client.4375.objecter successfully canceled tick 2024-03-25T19:45:17.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.936+0000 7efc1abb5640 1 -- 172.21.15.81:0/2565375073 >> v1:172.21.15.98:6800/3980829775 conn(0x7efbf806de00 legacy=0x7efbf80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.936+0000 7efc1abb5640 10 monclient: shutdown 2024-03-25T19:45:17.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.936+0000 7efc1abb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:17.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.936+0000 7efc1abb5640 1 -- 172.21.15.81:0/2565375073 >> v1:172.21.15.98:6789/0 conn(0x7efc1414be90 legacy=0x7efc141e3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.936+0000 7efc1abb5640 1 -- 172.21.15.81:0/2565375073 shutdown_connections 2024-03-25T19:45:17.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.936+0000 7efc1abb5640 1 -- 172.21.15.81:0/2565375073 >> 172.21.15.81:0/2565375073 conn(0x7efc140617a0 msgr2=0x7efc1413c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:17.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.936+0000 7efc1abb5640 1 -- 172.21.15.81:0/2565375073 shutdown_connections 2024-03-25T19:45:17.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.936+0000 7efc1abb5640 1 -- 172.21.15.81:0/2565375073 wait complete. 2024-03-25T19:45:17.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:17.936+0000 7efc1abb5640 1 librados: shutdown 2024-03-25T19:45:17.948 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:24.757 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:24.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.875+0000 7f35c2e82640 10 monclient: get_monmap_and_config 2024-03-25T19:45:24.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.875+0000 7f35c2e82640 10 monclient: build_initial_monmap 2024-03-25T19:45:24.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.875+0000 7f35c2e82640 10 monclient: monmap: 2024-03-25T19:45:24.875 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:45:24.875 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:24.875 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:45:24.876290+0000 2024-03-25T19:45:24.875 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:45:24.876290+0000 2024-03-25T19:45:24.875 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:24.875 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:24.875 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:45:24.876 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:45:24.876 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:45:24.876 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:24.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.876+0000 7f35c2e82640 1 Processor -- start 2024-03-25T19:45:24.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.876+0000 7f35c2e82640 1 -- start start 2024-03-25T19:45:24.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.876+0000 7f35c2e82640 10 monclient: init 2024-03-25T19:45:24.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.877+0000 7f35c2e82640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:24.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.877+0000 7f35c2e82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:24.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.877+0000 7f35c2e82640 10 monclient(hunting): picked mon.noname-a con 0x7f35bc0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:45:24.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.877+0000 7f35c2e82640 10 monclient(hunting): picked mon.noname-b con 0x7f35bc13f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:45:24.878 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.877+0000 7f35c2e82640 10 monclient(hunting): picked mon.noname-c con 0x7f35bc1430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:45:24.878 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.877+0000 7f35c2e82640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc142520 con 0x7f35bc0a98f0 2024-03-25T19:45:24.878 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.877+0000 7f35c2e82640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc0b28a0 con 0x7f35bc1430c0 2024-03-25T19:45:24.878 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.877+0000 7f35c2e82640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc0b2a30 con 0x7f35bc13f710 2024-03-25T19:45:24.878 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.877+0000 7f35c2e82640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:24.878 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.877+0000 7f35c2e82640 10 monclient(hunting): authenticate will time out at 4159.959364s 2024-03-25T19:45:24.882 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.883+0000 7f35c0c1e640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f35bc0a98f0 0x7f35bc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:35120/0 (socket says 172.21.15.81:35120) 2024-03-25T19:45:24.882 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.883+0000 7f35c0c1e640 1 -- 172.21.15.81:0/3928399484 learned_addr learned my addr 172.21.15.81:0/3928399484 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:45:24.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.888+0000 7f35bb7fe640 1 -- 172.21.15.81:0/3928399484 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207379332 0 0) 0x7f35bc142520 con 0x7f35bc0a98f0 2024-03-25T19:45:24.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.888+0000 7f35bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:24.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.888+0000 7f35bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:24.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.888+0000 7f35bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:24.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.888+0000 7f35bb7fe640 10 monclient(hunting): my global_id is 4380 2024-03-25T19:45:24.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.888+0000 7f35bb7fe640 1 -- 172.21.15.81:0/3928399484 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35ac003520 con 0x7f35bc0a98f0 2024-03-25T19:45:24.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.898+0000 7f35bb7fe640 1 -- 172.21.15.81:0/3928399484 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2316576051 0 0) 0x7f35ac003520 con 0x7f35bc0a98f0 2024-03-25T19:45:24.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.898+0000 7f35bb7fe640 1 -- 172.21.15.81:0/3928399484 >> v1:172.21.15.81:6790/0 conn(0x7f35bc13f710 legacy=0x7f35bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:24.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.898+0000 7f35bb7fe640 1 -- 172.21.15.81:0/3928399484 >> v1:172.21.15.98:6789/0 conn(0x7f35bc1430c0 legacy=0x7f35bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:24.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.898+0000 7f35bb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:24.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.898+0000 7f35bb7fe640 1 monclient: found mon.noname-a 2024-03-25T19:45:24.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.898+0000 7f35bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:24.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.898+0000 7f35bb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:45:24.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.898+0000 7f35bb7fe640 1 -- 172.21.15.81:0/3928399484 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc0b44c0 con 0x7f35bc0a98f0 2024-03-25T19:45:24.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.898+0000 7f35bb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:24.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.898+0000 7f35bb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:24.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.898+0000 7f35bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:24.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.898+0000 7f35bb7fe640 1 -- 172.21.15.81:0/3928399484 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f35b0001690 con 0x7f35bc0a98f0 2024-03-25T19:45:24.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.898+0000 7f35bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:24.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.898+0000 7f35bb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:24.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.898+0000 7f35bb7fe640 10 monclient: dump: 2024-03-25T19:45:24.899 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:24.899 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:24.899 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:24.899 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:24.899 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:24.899 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:24.899 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:24.899 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:24.899 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:24.900 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:24.900 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.898+0000 7f35c2e82640 5 monclient: authenticate success, global_id 4380 2024-03-25T19:45:24.900 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.898+0000 7f35c2e82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:24.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.902+0000 7f35bb7fe640 1 -- 172.21.15.81:0/3928399484 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35b00017f0 con 0x7f35bc0a98f0 2024-03-25T19:45:24.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.902+0000 7f35bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:24.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.902+0000 7f35bb7fe640 1 -- 172.21.15.81:0/3928399484 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f35b0001c00 con 0x7f35bc0a98f0 2024-03-25T19:45:24.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.902+0000 7f35c2e82640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:24.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.902+0000 7f35c2e82640 4 set_mon_vals no callback set 2024-03-25T19:45:24.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.902+0000 7f35bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:24.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.902+0000 7f35bb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:24.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.902+0000 7f35bb7fe640 10 monclient: dump: 2024-03-25T19:45:24.903 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:24.903 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:24.903 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:24.903 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:24.903 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:24.903 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:24.903 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:24.903 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:24.903 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:24.903 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:24.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.903+0000 7f35c2e82640 10 monclient: shutdown 2024-03-25T19:45:24.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.903+0000 7f35c2e82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:24.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.903+0000 7f35c2e82640 1 -- 172.21.15.81:0/3928399484 >> v1:172.21.15.81:6789/0 conn(0x7f35bc0a98f0 legacy=0x7f35bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:24.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.903+0000 7f35c2e82640 1 -- 172.21.15.81:0/3928399484 shutdown_connections 2024-03-25T19:45:24.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.903+0000 7f35c2e82640 1 -- 172.21.15.81:0/3928399484 >> 172.21.15.81:0/3928399484 conn(0x7f35bc0b7ea0 msgr2=0x7f35bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:24.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.904+0000 7f35c2e82640 1 -- 172.21.15.81:0/3928399484 shutdown_connections 2024-03-25T19:45:24.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.904+0000 7f35c2e82640 1 -- 172.21.15.81:0/3928399484 wait complete. 2024-03-25T19:45:24.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.904+0000 7f35c2e82640 10 monclient: build_initial_monmap 2024-03-25T19:45:24.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.904+0000 7f35c2e82640 10 monclient: monmap: 2024-03-25T19:45:24.905 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:45:24.905 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:24.905 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:45:24.905 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:45:24.905 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:24.905 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:45:24.905 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:45:24.906 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:45:24.906 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:45:24.906 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:24.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.904+0000 7f35c2e82640 1 librados: starting msgr at 2024-03-25T19:45:24.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.904+0000 7f35c2e82640 1 librados: starting objecter 2024-03-25T19:45:24.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.905+0000 7f35c2e82640 1 Processor -- start 2024-03-25T19:45:24.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.905+0000 7f35c2e82640 1 -- start start 2024-03-25T19:45:24.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.905+0000 7f35c2e82640 1 librados: setting wanted keys 2024-03-25T19:45:24.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.905+0000 7f35c2e82640 1 librados: calling monclient init 2024-03-25T19:45:24.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.905+0000 7f35c2e82640 10 monclient: init 2024-03-25T19:45:24.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.905+0000 7f35c2e82640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:24.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.905+0000 7f35c2e82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:24.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.905+0000 7f35c2e82640 10 monclient(hunting): picked mon.noname-b con 0x7f35bc0a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:45:24.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.905+0000 7f35c2e82640 10 monclient(hunting): picked mon.noname-c con 0x7f35bc13f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:45:24.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.905+0000 7f35c2e82640 10 monclient(hunting): picked mon.noname-a con 0x7f35bc1430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:45:24.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.905+0000 7f35c2e82640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc138c70 con 0x7f35bc1430c0 2024-03-25T19:45:24.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.905+0000 7f35c2e82640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc1367d0 con 0x7f35bc0a98f0 2024-03-25T19:45:24.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.905+0000 7f35c2e82640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc136960 con 0x7f35bc13f710 2024-03-25T19:45:24.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.905+0000 7f35c2e82640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:24.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.905+0000 7f35c2e82640 10 monclient(hunting): authenticate will time out at 4159.987823s 2024-03-25T19:45:24.922 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.923+0000 7f35bbfff640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f35bc13f710 0x7f35bc132f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:49800/0 (socket says 172.21.15.81:49800) 2024-03-25T19:45:24.922 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.923+0000 7f35bbfff640 1 -- 172.21.15.81:0/3809162913 learned_addr learned my addr 172.21.15.81:0/3809162913 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:45:24.923 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.923+0000 7f35b97fa640 10 client.?.objecter ms_handle_connect 0x7f35bc13f710 2024-03-25T19:45:24.923 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.923+0000 7f35b97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:24.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.928+0000 7f35b97fa640 10 client.?.objecter ms_handle_connect 0x7f35bc0a98f0 2024-03-25T19:45:24.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.928+0000 7f35b97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:24.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.932+0000 7f35b97fa640 10 client.?.objecter ms_handle_connect 0x7f35bc1430c0 2024-03-25T19:45:24.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.932+0000 7f35b97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:24.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.935+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286762506 0 0) 0x7f35bc136960 con 0x7f35bc13f710 2024-03-25T19:45:24.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.935+0000 7f35b97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:24.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.935+0000 7f35b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:24.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.935+0000 7f35b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:24.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.935+0000 7f35b97fa640 10 monclient(hunting): my global_id is 4367 2024-03-25T19:45:24.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.936+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35a0004eb0 con 0x7f35bc13f710 2024-03-25T19:45:24.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.937+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786670973 0 0) 0x7f35bc1367d0 con 0x7f35bc0a98f0 2024-03-25T19:45:24.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.937+0000 7f35b97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:24.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.937+0000 7f35b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:24.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.937+0000 7f35b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:24.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.937+0000 7f35b97fa640 10 monclient(hunting): my global_id is 4381 2024-03-25T19:45:24.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.937+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35bc136960 con 0x7f35bc0a98f0 2024-03-25T19:45:24.948 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.949+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020894569 0 0) 0x7f35bc138c70 con 0x7f35bc1430c0 2024-03-25T19:45:24.949 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.949+0000 7f35b97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:24.949 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.949+0000 7f35b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:24.949 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.949+0000 7f35b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:24.949 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.949+0000 7f35b97fa640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:45:24.949 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.949+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35bc1367d0 con 0x7f35bc1430c0 2024-03-25T19:45:24.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.950+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 277726745 0 0) 0x7f35bc136960 con 0x7f35bc0a98f0 2024-03-25T19:45:24.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.950+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35bc138c70 con 0x7f35bc0a98f0 2024-03-25T19:45:24.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.950+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f35b0001b00 con 0x7f35bc0a98f0 2024-03-25T19:45:24.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.950+0000 7f35b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:24.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.950+0000 7f35b97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:24.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.950+0000 7f35b97fa640 10 monclient(hunting): dump: 2024-03-25T19:45:24.950 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:24.950 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:24.950 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:24.950 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:24.950 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:24.951 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:24.951 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:24.951 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:24.951 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:24.951 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:24.951 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.950+0000 7f35b97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:24.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.953+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3085033588 0 0) 0x7f35a0004eb0 con 0x7f35bc13f710 2024-03-25T19:45:24.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.953+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35bc136960 con 0x7f35bc13f710 2024-03-25T19:45:24.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.953+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f35a8001a30 con 0x7f35bc13f710 2024-03-25T19:45:24.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.953+0000 7f35b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:24.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.953+0000 7f35b97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:24.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.953+0000 7f35b97fa640 10 monclient(hunting): dump: 2024-03-25T19:45:24.953 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:24.953 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:24.953 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:24.953 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:24.953 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:24.953 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:24.953 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:24.953 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:24.953 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:24.953 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:24.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 860619891 0 0) 0x7f35bc138c70 con 0x7f35bc0a98f0 2024-03-25T19:45:24.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 >> v1:172.21.15.81:6790/0 conn(0x7f35bc13f710 legacy=0x7f35bc132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:24.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 >> v1:172.21.15.81:6789/0 conn(0x7f35bc1430c0 legacy=0x7f35bc1e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:24.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35b97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:45:24.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35b97fa640 1 monclient: found mon.b 2024-03-25T19:45:24.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:24.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35b97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:24.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc139630 con 0x7f35bc0a98f0 2024-03-25T19:45:24.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35b97fa640 10 monclient: _finish_auth 0 2024-03-25T19:45:24.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35b97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:45:24.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:24.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35c2e82640 5 monclient: authenticate success, global_id 4381 2024-03-25T19:45:24.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35c2e82640 10 monclient: _renew_subs 2024-03-25T19:45:24.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35c2e82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:24.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35c2e82640 1 -- 172.21.15.81:0/3809162913 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35bc139880 con 0x7f35bc0a98f0 2024-03-25T19:45:24.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35c2e82640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:24.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35c2e82640 10 monclient: _renew_subs 2024-03-25T19:45:24.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35c2e82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:24.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35c2e82640 1 -- 172.21.15.81:0/3809162913 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35bc1337f0 con 0x7f35bc0a98f0 2024-03-25T19:45:24.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.965+0000 7f35c2e82640 1 librados: init done 2024-03-25T19:45:24.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.966+0000 7f35c2e82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:24.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.966+0000 7f35c2e82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:24.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.966+0000 7f35c2e82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:24.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:24.966+0000 7f35c2e82640 1 -- 172.21.15.81:0/3809162913 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35bc0627b0 con 0x7f35bc0a98f0 2024-03-25T19:45:25.241 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.241+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35b0001ca0 con 0x7f35bc0a98f0 2024-03-25T19:45:25.242 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.241+0000 7f35b97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:25.242 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.241+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f35b0004c30 con 0x7f35bc0a98f0 2024-03-25T19:45:25.242 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.241+0000 7f35b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:25.242 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.241+0000 7f35b97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:25.242 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.241+0000 7f35b97fa640 10 monclient: dump: 2024-03-25T19:45:25.242 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:25.242 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:25.242 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:25.243 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:25.243 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:25.243 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:25.243 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:25.243 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:25.243 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:25.243 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:25.243 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.241+0000 7f35baffd640 4 set_mon_vals no callback set 2024-03-25T19:45:25.243 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.242+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f35b001ac00 con 0x7f35bc0a98f0 2024-03-25T19:45:25.244 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.242+0000 7f35bbfff640 10 monclient: get_auth_request con 0x7f35a006df70 auth_method 0 2024-03-25T19:45:25.244 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.242+0000 7f35b97fa640 10 client.4381.objecter ms_handle_connect 0x7f35a006df70 2024-03-25T19:45:25.245 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.245+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f35b0054750 con 0x7f35bc0a98f0 2024-03-25T19:45:25.245 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.245+0000 7f35b97fa640 10 client.4381.objecter ms_dispatch 0x7f35bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:25.245 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.245+0000 7f35b97fa640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:25.245 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.245+0000 7f35b97fa640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:25.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.245+0000 7f35b97fa640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-25T19:45:25.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.248+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35b0050260 con 0x7f35bc0a98f0 2024-03-25T19:45:25.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.248+0000 7f35b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:25.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.248+0000 7f35b97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:25.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.505+0000 7f35c2e82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:25.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.505+0000 7f35c2e82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:25.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.505+0000 7f35c2e82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:25.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.505+0000 7f35c2e82640 1 -- 172.21.15.81:0/3809162913 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35bc074cc0 con 0x7f35bc0a98f0 2024-03-25T19:45:25.528 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.528+0000 7f35b97fa640 1 -- 172.21.15.81:0/3809162913 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f35b0052190 con 0x7f35bc0a98f0 2024-03-25T19:45:25.528 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.528+0000 7f35b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:25.528 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.528+0000 7f35b97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:25.528 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:45:25.528 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:25.531 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.531+0000 7f35c2e82640 10 librados: watch_flush enter 2024-03-25T19:45:25.531 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.531+0000 7f35c2e82640 10 librados: watch_flush exit 2024-03-25T19:45:25.531 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.531+0000 7f35c2e82640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-25T19:45:25.531 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.531+0000 7f35c2e82640 10 client.4381.objecter successfully canceled tick 2024-03-25T19:45:25.531 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.531+0000 7f35c2e82640 1 -- 172.21.15.81:0/3809162913 >> v1:172.21.15.98:6800/3980829775 conn(0x7f35a006df70 legacy=0x7f35a0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.531 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.531+0000 7f35c2e82640 10 monclient: shutdown 2024-03-25T19:45:25.531 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.531+0000 7f35c2e82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:25.531 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.531+0000 7f35c2e82640 1 -- 172.21.15.81:0/3809162913 >> v1:172.21.15.98:6789/0 conn(0x7f35bc0a98f0 legacy=0x7f35bc137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.531 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.531+0000 7f35c2e82640 1 -- 172.21.15.81:0/3809162913 shutdown_connections 2024-03-25T19:45:25.531 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.531+0000 7f35c2e82640 1 -- 172.21.15.81:0/3809162913 >> 172.21.15.81:0/3809162913 conn(0x7f35bc0b7ea0 msgr2=0x7f35bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:25.532 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.531+0000 7f35c2e82640 1 -- 172.21.15.81:0/3809162913 shutdown_connections 2024-03-25T19:45:25.532 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.531+0000 7f35c2e82640 1 -- 172.21.15.81:0/3809162913 wait complete. 2024-03-25T19:45:25.532 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:25.531+0000 7f35c2e82640 1 librados: shutdown 2024-03-25T19:45:25.542 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:32.350 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:32.467 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.467+0000 7f10ffd72640 10 monclient: get_monmap_and_config 2024-03-25T19:45:32.467 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.467+0000 7f10ffd72640 10 monclient: build_initial_monmap 2024-03-25T19:45:32.467 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.467+0000 7f10ffd72640 10 monclient: monmap: 2024-03-25T19:45:32.467 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:45:32.467 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:32.467 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:45:32.468015+0000 2024-03-25T19:45:32.467 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:45:32.468015+0000 2024-03-25T19:45:32.467 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:32.467 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:32.467 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:45:32.467 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:45:32.468 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:45:32.468 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:32.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.467+0000 7f10ffd72640 1 Processor -- start 2024-03-25T19:45:32.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.468+0000 7f10ffd72640 1 -- start start 2024-03-25T19:45:32.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.468+0000 7f10ffd72640 10 monclient: init 2024-03-25T19:45:32.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.469+0000 7f10ffd72640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:32.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.469+0000 7f10ffd72640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:32.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.469+0000 7f10ffd72640 10 monclient(hunting): picked mon.noname-a con 0x7f10f80a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:45:32.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.469+0000 7f10ffd72640 10 monclient(hunting): picked mon.noname-b con 0x7f10f813f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:45:32.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.469+0000 7f10ffd72640 10 monclient(hunting): picked mon.noname-c con 0x7f10f81430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:45:32.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.469+0000 7f10ffd72640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f8142520 con 0x7f10f80a98f0 2024-03-25T19:45:32.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.469+0000 7f10ffd72640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f80b28a0 con 0x7f10f81430c0 2024-03-25T19:45:32.470 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.469+0000 7f10ffd72640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f80b2a30 con 0x7f10f813f710 2024-03-25T19:45:32.470 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.469+0000 7f10ffd72640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:32.470 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.469+0000 7f10ffd72640 10 monclient(hunting): authenticate will time out at 4167.550984s 2024-03-25T19:45:32.483 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.483+0000 7f10fdb0e640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f10f80a98f0 0x7f10f80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:50896/0 (socket says 172.21.15.81:50896) 2024-03-25T19:45:32.483 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.483+0000 7f10fdb0e640 1 -- 172.21.15.81:0/3729487283 learned_addr learned my addr 172.21.15.81:0/3729487283 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:45:32.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.502+0000 7f10fcb0c640 1 -- 172.21.15.81:0/3729487283 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2105781706 0 0) 0x7f10f80b28a0 con 0x7f10f81430c0 2024-03-25T19:45:32.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.502+0000 7f10fcb0c640 10 monclient(hunting): _negotiate 2024-03-25T19:45:32.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.502+0000 7f10fcb0c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:32.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.502+0000 7f10fcb0c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:32.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.502+0000 7f10fcb0c640 10 monclient(hunting): my global_id is 4384 2024-03-25T19:45:32.502 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.502+0000 7f10fcb0c640 1 -- 172.21.15.81:0/3729487283 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10e4003520 con 0x7f10f81430c0 2024-03-25T19:45:32.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.504+0000 7f10fcb0c640 1 -- 172.21.15.81:0/3729487283 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806056194 0 0) 0x7f10f8142520 con 0x7f10f80a98f0 2024-03-25T19:45:32.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.504+0000 7f10fcb0c640 10 monclient(hunting): _negotiate 2024-03-25T19:45:32.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.504+0000 7f10fcb0c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:32.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.504+0000 7f10fcb0c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:32.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.504+0000 7f10fcb0c640 10 monclient(hunting): my global_id is 4386 2024-03-25T19:45:32.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.504+0000 7f10fcb0c640 1 -- 172.21.15.81:0/3729487283 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f80b28a0 con 0x7f10f80a98f0 2024-03-25T19:45:32.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.506+0000 7f10fcb0c640 1 -- 172.21.15.81:0/3729487283 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844045507 0 0) 0x7f10f80b2a30 con 0x7f10f813f710 2024-03-25T19:45:32.506 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.506+0000 7f10fcb0c640 10 monclient(hunting): _negotiate 2024-03-25T19:45:32.506 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.506+0000 7f10fcb0c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:32.506 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.506+0000 7f10fcb0c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:32.506 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.506+0000 7f10fcb0c640 10 monclient(hunting): my global_id is 4370 2024-03-25T19:45:32.506 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.506+0000 7f10fcb0c640 1 -- 172.21.15.81:0/3729487283 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f8142520 con 0x7f10f813f710 2024-03-25T19:45:32.526 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.526+0000 7f10fcb0c640 1 -- 172.21.15.81:0/3729487283 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 682136199 0 0) 0x7f10e4003520 con 0x7f10f81430c0 2024-03-25T19:45:32.526 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.526+0000 7f10fcb0c640 1 -- 172.21.15.81:0/3729487283 >> v1:172.21.15.81:6790/0 conn(0x7f10f813f710 legacy=0x7f10f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:32.526 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.526+0000 7f10fcb0c640 1 -- 172.21.15.81:0/3729487283 >> v1:172.21.15.81:6789/0 conn(0x7f10f80a98f0 legacy=0x7f10f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:32.526 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.526+0000 7f10fcb0c640 10 monclient: _finish_hunting 0 2024-03-25T19:45:32.526 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.526+0000 7f10fcb0c640 1 monclient: found mon.noname-c 2024-03-25T19:45:32.526 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.526+0000 7f10fcb0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:32.526 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.526+0000 7f10fcb0c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:45:32.526 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.526+0000 7f10fcb0c640 1 -- 172.21.15.81:0/3729487283 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f80b44c0 con 0x7f10f81430c0 2024-03-25T19:45:32.526 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.526+0000 7f10fcb0c640 10 monclient: _finish_auth 0 2024-03-25T19:45:32.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.526+0000 7f10fcb0c640 10 monclient: _check_auth_tickets 2024-03-25T19:45:32.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.526+0000 7f10fcb0c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:32.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.526+0000 7f10fcb0c640 1 -- 172.21.15.81:0/3729487283 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f10f0001690 con 0x7f10f81430c0 2024-03-25T19:45:32.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.526+0000 7f10fcb0c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:32.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.526+0000 7f10fcb0c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:32.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.526+0000 7f10fcb0c640 10 monclient: dump: 2024-03-25T19:45:32.527 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:32.527 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:32.527 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:32.527 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:32.527 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:32.527 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:32.527 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:32.527 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:32.527 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:32.528 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:32.528 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.526+0000 7f10ffd72640 5 monclient: authenticate success, global_id 4384 2024-03-25T19:45:32.528 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.526+0000 7f10ffd72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:32.554 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.554+0000 7f10fcb0c640 1 -- 172.21.15.81:0/3729487283 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10f00017f0 con 0x7f10f81430c0 2024-03-25T19:45:32.554 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.554+0000 7f10fcb0c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:32.554 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.554+0000 7f10fcb0c640 1 -- 172.21.15.81:0/3729487283 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f10f0001c00 con 0x7f10f81430c0 2024-03-25T19:45:32.554 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.554+0000 7f10fcb0c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:32.554 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.554+0000 7f10fcb0c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:32.554 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.554+0000 7f10fcb0c640 10 monclient: dump: 2024-03-25T19:45:32.554 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:32.554 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:32.554 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:32.554 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:32.555 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:32.555 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:32.555 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:32.555 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:32.555 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:32.555 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:32.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.554+0000 7f10ffd72640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:32.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.554+0000 7f10ffd72640 4 set_mon_vals no callback set 2024-03-25T19:45:32.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.555+0000 7f10ffd72640 10 monclient: shutdown 2024-03-25T19:45:32.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.555+0000 7f10ffd72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:32.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.555+0000 7f10ffd72640 1 -- 172.21.15.81:0/3729487283 >> v1:172.21.15.98:6789/0 conn(0x7f10f81430c0 legacy=0x7f10f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:32.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.555+0000 7f10ffd72640 1 -- 172.21.15.81:0/3729487283 shutdown_connections 2024-03-25T19:45:32.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.555+0000 7f10ffd72640 1 -- 172.21.15.81:0/3729487283 >> 172.21.15.81:0/3729487283 conn(0x7f10f80b7ea0 msgr2=0x7f10f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:32.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.555+0000 7f10ffd72640 1 -- 172.21.15.81:0/3729487283 shutdown_connections 2024-03-25T19:45:32.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.555+0000 7f10ffd72640 1 -- 172.21.15.81:0/3729487283 wait complete. 2024-03-25T19:45:32.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.556+0000 7f10ffd72640 10 monclient: build_initial_monmap 2024-03-25T19:45:32.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.556+0000 7f10ffd72640 10 monclient: monmap: 2024-03-25T19:45:32.558 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:45:32.558 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:32.558 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:45:32.558 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:45:32.558 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:32.558 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:45:32.558 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:45:32.558 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:45:32.558 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:45:32.558 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:32.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.556+0000 7f10ffd72640 1 librados: starting msgr at 2024-03-25T19:45:32.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.556+0000 7f10ffd72640 1 librados: starting objecter 2024-03-25T19:45:32.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.556+0000 7f10ffd72640 1 Processor -- start 2024-03-25T19:45:32.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.556+0000 7f10ffd72640 1 -- start start 2024-03-25T19:45:32.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.556+0000 7f10ffd72640 1 librados: setting wanted keys 2024-03-25T19:45:32.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.556+0000 7f10ffd72640 1 librados: calling monclient init 2024-03-25T19:45:32.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.556+0000 7f10ffd72640 10 monclient: init 2024-03-25T19:45:32.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.557+0000 7f10ffd72640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:32.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.557+0000 7f10ffd72640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:32.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.557+0000 7f10ffd72640 10 monclient(hunting): picked mon.noname-c con 0x7f10f80a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:45:32.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.557+0000 7f10ffd72640 10 monclient(hunting): picked mon.noname-b con 0x7f10f813f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:45:32.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.557+0000 7f10ffd72640 10 monclient(hunting): picked mon.noname-a con 0x7f10f81430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:45:32.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.557+0000 7f10ffd72640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f81369d0 con 0x7f10f81430c0 2024-03-25T19:45:32.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.557+0000 7f10ffd72640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f8133a60 con 0x7f10f813f710 2024-03-25T19:45:32.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.557+0000 7f10ffd72640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f80ab9e0 con 0x7f10f80a98f0 2024-03-25T19:45:32.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.557+0000 7f10ffd72640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:32.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.557+0000 7f10ffd72640 10 monclient(hunting): authenticate will time out at 4167.639455s 2024-03-25T19:45:32.574 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.574+0000 7f10fd30d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f10f813f710 0x7f10f81362c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:43102/0 (socket says 172.21.15.81:43102) 2024-03-25T19:45:32.574 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.574+0000 7f10fd30d640 1 -- 172.21.15.81:0/8179576 learned_addr learned my addr 172.21.15.81:0/8179576 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:45:32.577 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.578+0000 7f10ee7fc640 10 client.?.objecter ms_handle_connect 0x7f10f813f710 2024-03-25T19:45:32.578 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.578+0000 7f10ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:32.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.583+0000 7f10ee7fc640 10 client.?.objecter ms_handle_connect 0x7f10f80a98f0 2024-03-25T19:45:32.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.583+0000 7f10ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:32.584 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.585+0000 7f10ee7fc640 10 client.?.objecter ms_handle_connect 0x7f10f81430c0 2024-03-25T19:45:32.584 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.585+0000 7f10ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:32.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.589+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073338780 0 0) 0x7f10f8133a60 con 0x7f10f813f710 2024-03-25T19:45:32.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.589+0000 7f10ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:32.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.589+0000 7f10ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:32.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.589+0000 7f10ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:32.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.589+0000 7f10ee7fc640 10 monclient(hunting): my global_id is 4387 2024-03-25T19:45:32.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.589+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10dc004eb0 con 0x7f10f813f710 2024-03-25T19:45:32.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.607+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461670975 0 0) 0x7f10f80ab9e0 con 0x7f10f80a98f0 2024-03-25T19:45:32.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.607+0000 7f10ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:32.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.607+0000 7f10ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:32.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.607+0000 7f10ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:32.607 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.607+0000 7f10ee7fc640 10 monclient(hunting): my global_id is 4373 2024-03-25T19:45:32.607 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.607+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f8133a60 con 0x7f10f80a98f0 2024-03-25T19:45:32.617 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.617+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432205488 0 0) 0x7f10f81369d0 con 0x7f10f81430c0 2024-03-25T19:45:32.617 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.617+0000 7f10ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:32.617 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.617+0000 7f10ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:32.617 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.617+0000 7f10ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:32.617 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.617+0000 7f10ee7fc640 10 monclient(hunting): my global_id is 4389 2024-03-25T19:45:32.617 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.617+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f80ab9e0 con 0x7f10f81430c0 2024-03-25T19:45:32.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.624+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1172032138 0 0) 0x7f10dc004eb0 con 0x7f10f813f710 2024-03-25T19:45:32.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.624+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10f81369d0 con 0x7f10f813f710 2024-03-25T19:45:32.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.624+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f10e8001a10 con 0x7f10f813f710 2024-03-25T19:45:32.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.624+0000 7f10ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:32.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.625+0000 7f10ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:32.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.625+0000 7f10ee7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:32.624 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:32.625 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:32.625 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:32.625 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:32.625 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:32.625 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:32.625 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:32.625 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:32.625 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:32.625 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:32.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.625+0000 7f10ee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:32.634 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.635+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1792714559 0 0) 0x7f10f8133a60 con 0x7f10f80a98f0 2024-03-25T19:45:32.634 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.635+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10dc004eb0 con 0x7f10f80a98f0 2024-03-25T19:45:32.634 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.635+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f10f4001cf0 con 0x7f10f80a98f0 2024-03-25T19:45:32.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.635+0000 7f10ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:32.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.635+0000 7f10ee7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:32.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.635+0000 7f10ee7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:32.635 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:32.635 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:32.635 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:32.635 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:32.635 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:32.635 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:32.635 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:32.635 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:32.635 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:32.635 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:32.639 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3019039431 0 0) 0x7f10f81369d0 con 0x7f10f813f710 2024-03-25T19:45:32.639 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 >> v1:172.21.15.81:6790/0 conn(0x7f10f80a98f0 legacy=0x7f10f8132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:32.639 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 >> v1:172.21.15.81:6789/0 conn(0x7f10f81430c0 legacy=0x7f10f8133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:32.639 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:32.639 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ee7fc640 1 monclient: found mon.b 2024-03-25T19:45:32.639 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:32.639 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ee7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:32.639 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f80abcc0 con 0x7f10f813f710 2024-03-25T19:45:32.639 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ee7fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:32.639 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:32.639 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:32.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ffd72640 5 monclient: authenticate success, global_id 4387 2024-03-25T19:45:32.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ffd72640 10 monclient: _renew_subs 2024-03-25T19:45:32.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ffd72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:32.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ffd72640 1 -- 172.21.15.81:0/8179576 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f80abfa0 con 0x7f10f813f710 2024-03-25T19:45:32.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ffd72640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:32.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ffd72640 10 monclient: _renew_subs 2024-03-25T19:45:32.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ffd72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:32.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ffd72640 1 -- 172.21.15.81:0/8179576 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f8137e40 con 0x7f10f813f710 2024-03-25T19:45:32.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.639+0000 7f10ffd72640 1 librados: init done 2024-03-25T19:45:32.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.640+0000 7f10ffd72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:32.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.640+0000 7f10ffd72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:32.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.640+0000 7f10ffd72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:32.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.640+0000 7f10ffd72640 1 -- 172.21.15.81:0/8179576 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f80627b0 con 0x7f10f813f710 2024-03-25T19:45:32.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.967+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10e8001c70 con 0x7f10f813f710 2024-03-25T19:45:32.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.967+0000 7f10ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:32.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.967+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f10e8003a60 con 0x7f10f813f710 2024-03-25T19:45:32.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.967+0000 7f10ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:32.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.967+0000 7f10ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:32.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.967+0000 7f10ee7fc640 10 monclient: dump: 2024-03-25T19:45:32.967 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:32.967 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:32.967 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:32.967 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:32.968 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:32.968 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:32.968 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:32.968 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:32.968 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:32.968 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:32.968 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.967+0000 7f10effff640 4 set_mon_vals no callback set 2024-03-25T19:45:32.968 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.968+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f10e8003c80 con 0x7f10f813f710 2024-03-25T19:45:32.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.968+0000 7f10fdb0e640 10 monclient: get_auth_request con 0x7f10dc06df70 auth_method 0 2024-03-25T19:45:32.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.969+0000 7f10ee7fc640 10 client.4387.objecter ms_handle_connect 0x7f10dc06df70 2024-03-25T19:45:32.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.973+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f10e80548d0 con 0x7f10f813f710 2024-03-25T19:45:32.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.973+0000 7f10ee7fc640 10 client.4387.objecter ms_dispatch 0x7f10f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:32.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.973+0000 7f10ee7fc640 3 client.4387.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:32.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.973+0000 7f10ee7fc640 3 client.4387.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:32.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.973+0000 7f10ee7fc640 20 client.4387.objecter dump_active .. 0 homeless 2024-03-25T19:45:32.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.974+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f10e80503e0 con 0x7f10f813f710 2024-03-25T19:45:32.974 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.974+0000 7f10ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:32.974 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:32.974+0000 7f10ee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:33.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.226+0000 7f10ffd72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:33.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.226+0000 7f10ffd72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:33.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.226+0000 7f10ffd72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:33.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.226+0000 7f10ffd72640 1 -- 172.21.15.81:0/8179576 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10f8139000 con 0x7f10f813f710 2024-03-25T19:45:33.272 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.272+0000 7f10ee7fc640 1 -- 172.21.15.81:0/8179576 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f10e8052310 con 0x7f10f813f710 2024-03-25T19:45:33.272 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.272+0000 7f10ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:33.272 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.272+0000 7f10ee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:33.272 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:45:33.272 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:33.275 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.274+0000 7f10ffd72640 10 librados: watch_flush enter 2024-03-25T19:45:33.275 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.274+0000 7f10ffd72640 10 librados: watch_flush exit 2024-03-25T19:45:33.275 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.274+0000 7f10ffd72640 20 client.4387.objecter shutdown clearing up homeless session... 2024-03-25T19:45:33.275 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.274+0000 7f10ffd72640 10 client.4387.objecter successfully canceled tick 2024-03-25T19:45:33.275 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.274+0000 7f10ffd72640 1 -- 172.21.15.81:0/8179576 >> v1:172.21.15.98:6800/3980829775 conn(0x7f10dc06df70 legacy=0x7f10dc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.275 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.274+0000 7f10ffd72640 10 monclient: shutdown 2024-03-25T19:45:33.275 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.274+0000 7f10ffd72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:33.275 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.274+0000 7f10ffd72640 1 -- 172.21.15.81:0/8179576 >> v1:172.21.15.98:6789/0 conn(0x7f10f813f710 legacy=0x7f10f81362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.275 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.275+0000 7f10ffd72640 1 -- 172.21.15.81:0/8179576 shutdown_connections 2024-03-25T19:45:33.275 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.275+0000 7f10ffd72640 1 -- 172.21.15.81:0/8179576 >> 172.21.15.81:0/8179576 conn(0x7f10f80b7ea0 msgr2=0x7f10f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:33.275 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.275+0000 7f10ffd72640 1 -- 172.21.15.81:0/8179576 shutdown_connections 2024-03-25T19:45:33.275 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.275+0000 7f10ffd72640 1 -- 172.21.15.81:0/8179576 wait complete. 2024-03-25T19:45:33.275 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:33.275+0000 7f10ffd72640 1 librados: shutdown 2024-03-25T19:45:33.286 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:37.142 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:03:59.783 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:37.186 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:59.820 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:37.187 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:59.821 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:37.193 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:03:59.834 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:37.221 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:03:59.855 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:37.238 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:03:59.880 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:37.270 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:03:59.912 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:37.284 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:03:59.926 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:37.318 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:03:59.960 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:40.093 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:40.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.208+0000 7fef0f5cf640 10 monclient: get_monmap_and_config 2024-03-25T19:45:40.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.208+0000 7fef0f5cf640 10 monclient: build_initial_monmap 2024-03-25T19:45:40.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.208+0000 7fef0f5cf640 10 monclient: monmap: 2024-03-25T19:45:40.208 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:45:40.208 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:40.208 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:45:40.209453+0000 2024-03-25T19:45:40.208 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:45:40.209453+0000 2024-03-25T19:45:40.208 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:40.209 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:40.209 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:45:40.209 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:45:40.209 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:45:40.209 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:40.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.209+0000 7fef0f5cf640 1 Processor -- start 2024-03-25T19:45:40.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.209+0000 7fef0f5cf640 1 -- start start 2024-03-25T19:45:40.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.209+0000 7fef0f5cf640 10 monclient: init 2024-03-25T19:45:40.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.210+0000 7fef0f5cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:40.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.210+0000 7fef0f5cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:40.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.210+0000 7fef0f5cf640 10 monclient(hunting): picked mon.noname-a con 0x7fef080a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:45:40.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.210+0000 7fef0f5cf640 10 monclient(hunting): picked mon.noname-c con 0x7fef0813f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:45:40.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.210+0000 7fef0f5cf640 10 monclient(hunting): picked mon.noname-b con 0x7fef081430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:45:40.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.210+0000 7fef0f5cf640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef08142520 con 0x7fef080a98f0 2024-03-25T19:45:40.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.210+0000 7fef0f5cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef080b28a0 con 0x7fef0813f710 2024-03-25T19:45:40.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.210+0000 7fef0f5cf640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef080b2a30 con 0x7fef081430c0 2024-03-25T19:45:40.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.210+0000 7fef0f5cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:40.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.210+0000 7fef0f5cf640 10 monclient(hunting): authenticate will time out at 4175.292173s 2024-03-25T19:45:40.216 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.216+0000 7fef0d36b640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fef080a98f0 0x7fef080a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38080/0 (socket says 172.21.15.81:38080) 2024-03-25T19:45:40.216 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.216+0000 7fef0d36b640 1 -- 172.21.15.81:0/3795726966 learned_addr learned my addr 172.21.15.81:0/3795726966 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:45:40.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.228+0000 7feefffff640 1 -- 172.21.15.81:0/3795726966 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001505016 0 0) 0x7fef080b28a0 con 0x7fef0813f710 2024-03-25T19:45:40.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.228+0000 7feefffff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.228+0000 7feefffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.228+0000 7feefffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.228+0000 7feefffff640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:45:40.228 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.228+0000 7feefffff640 1 -- 172.21.15.81:0/3795726966 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feef4003520 con 0x7fef0813f710 2024-03-25T19:45:40.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.238+0000 7feefffff640 1 -- 172.21.15.81:0/3795726966 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245115448 0 0) 0x7fef08142520 con 0x7fef080a98f0 2024-03-25T19:45:40.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.238+0000 7feefffff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.238+0000 7feefffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.238+0000 7feefffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.238+0000 7feefffff640 10 monclient(hunting): my global_id is 4392 2024-03-25T19:45:40.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.238+0000 7feefffff640 1 -- 172.21.15.81:0/3795726966 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef080b28a0 con 0x7fef080a98f0 2024-03-25T19:45:40.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 1 -- 172.21.15.81:0/3795726966 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559334745 0 0) 0x7fef080b2a30 con 0x7fef081430c0 2024-03-25T19:45:40.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 10 monclient(hunting): my global_id is 4376 2024-03-25T19:45:40.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 1 -- 172.21.15.81:0/3795726966 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef08142520 con 0x7fef081430c0 2024-03-25T19:45:40.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 1 -- 172.21.15.81:0/3795726966 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3255381936 0 0) 0x7feef4003520 con 0x7fef0813f710 2024-03-25T19:45:40.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 1 -- 172.21.15.81:0/3795726966 >> v1:172.21.15.81:6790/0 conn(0x7fef081430c0 legacy=0x7fef0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 1 -- 172.21.15.81:0/3795726966 >> v1:172.21.15.81:6789/0 conn(0x7fef080a98f0 legacy=0x7fef080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 10 monclient: _finish_hunting 0 2024-03-25T19:45:40.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 1 monclient: found mon.noname-c 2024-03-25T19:45:40.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:40.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:45:40.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 1 -- 172.21.15.81:0/3795726966 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef080b44c0 con 0x7fef0813f710 2024-03-25T19:45:40.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 10 monclient: _finish_auth 0 2024-03-25T19:45:40.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 10 monclient: _check_auth_tickets 2024-03-25T19:45:40.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:40.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 1 -- 172.21.15.81:0/3795726966 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7feef8001690 con 0x7fef0813f710 2024-03-25T19:45:40.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:40.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:40.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7feefffff640 10 monclient: dump: 2024-03-25T19:45:40.240 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:40.240 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:40.240 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:40.240 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:40.240 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:40.240 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:40.240 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:40.240 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:40.240 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:40.241 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:40.241 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7fef0f5cf640 5 monclient: authenticate success, global_id 4390 2024-03-25T19:45:40.241 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.239+0000 7fef0f5cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:40.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.248+0000 7feefffff640 1 -- 172.21.15.81:0/3795726966 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feef8001870 con 0x7fef0813f710 2024-03-25T19:45:40.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.248+0000 7feefffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:40.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.248+0000 7feefffff640 1 -- 172.21.15.81:0/3795726966 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7feef8001c80 con 0x7fef0813f710 2024-03-25T19:45:40.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.248+0000 7feefffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:40.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.248+0000 7feefffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:40.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.248+0000 7feefffff640 10 monclient: dump: 2024-03-25T19:45:40.248 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:40.248 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:40.248 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:40.249 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:40.249 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:40.249 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:40.249 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:40.249 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:40.249 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:40.249 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:40.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.248+0000 7fef0f5cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:40.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.248+0000 7fef0f5cf640 4 set_mon_vals no callback set 2024-03-25T19:45:40.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.249+0000 7fef0f5cf640 10 monclient: shutdown 2024-03-25T19:45:40.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.249+0000 7fef0f5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:40.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.249+0000 7fef0f5cf640 1 -- 172.21.15.81:0/3795726966 >> v1:172.21.15.98:6789/0 conn(0x7fef0813f710 legacy=0x7fef081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.249+0000 7fef0f5cf640 1 -- 172.21.15.81:0/3795726966 shutdown_connections 2024-03-25T19:45:40.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.249+0000 7fef0f5cf640 1 -- 172.21.15.81:0/3795726966 >> 172.21.15.81:0/3795726966 conn(0x7fef080b7ea0 msgr2=0x7fef080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:40.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.249+0000 7fef0f5cf640 1 -- 172.21.15.81:0/3795726966 shutdown_connections 2024-03-25T19:45:40.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.249+0000 7fef0f5cf640 1 -- 172.21.15.81:0/3795726966 wait complete. 2024-03-25T19:45:40.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.250+0000 7fef0f5cf640 10 monclient: build_initial_monmap 2024-03-25T19:45:40.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.250+0000 7fef0f5cf640 10 monclient: monmap: 2024-03-25T19:45:40.252 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:45:40.252 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:40.252 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:45:40.252 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:45:40.252 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:40.253 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:45:40.253 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:45:40.253 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:45:40.253 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:45:40.253 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:40.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.251+0000 7fef0f5cf640 1 librados: starting msgr at 2024-03-25T19:45:40.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.251+0000 7fef0f5cf640 1 librados: starting objecter 2024-03-25T19:45:40.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.251+0000 7fef0f5cf640 1 Processor -- start 2024-03-25T19:45:40.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.251+0000 7fef0f5cf640 1 -- start start 2024-03-25T19:45:40.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.251+0000 7fef0f5cf640 1 librados: setting wanted keys 2024-03-25T19:45:40.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.251+0000 7fef0f5cf640 1 librados: calling monclient init 2024-03-25T19:45:40.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.251+0000 7fef0f5cf640 10 monclient: init 2024-03-25T19:45:40.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.252+0000 7fef0f5cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:40.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.252+0000 7fef0f5cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:40.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.252+0000 7fef0f5cf640 10 monclient(hunting): picked mon.noname-a con 0x7fef080a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:45:40.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.252+0000 7fef0f5cf640 10 monclient(hunting): picked mon.noname-b con 0x7fef0813f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:45:40.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.252+0000 7fef0f5cf640 10 monclient(hunting): picked mon.noname-c con 0x7fef081430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:45:40.254 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.252+0000 7fef0f5cf640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef080b7950 con 0x7fef080a98f0 2024-03-25T19:45:40.254 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.252+0000 7fef0f5cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0806fa50 con 0x7fef0813f710 2024-03-25T19:45:40.254 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.252+0000 7fef0f5cf640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef080b73c0 con 0x7fef081430c0 2024-03-25T19:45:40.254 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.252+0000 7fef0f5cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:40.254 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.252+0000 7fef0f5cf640 10 monclient(hunting): authenticate will time out at 4175.334141s 2024-03-25T19:45:40.255 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.255+0000 7fef0cb6a640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fef0813f710 0x7fef081d3120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:55960/0 (socket says 172.21.15.81:55960) 2024-03-25T19:45:40.255 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.255+0000 7fef0cb6a640 1 -- 172.21.15.81:0/3955643339 learned_addr learned my addr 172.21.15.81:0/3955643339 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:45:40.256 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.257+0000 7feefdffb640 10 client.?.objecter ms_handle_connect 0x7fef0813f710 2024-03-25T19:45:40.256 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.257+0000 7feefdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:40.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.263+0000 7feefdffb640 10 client.?.objecter ms_handle_connect 0x7fef080a98f0 2024-03-25T19:45:40.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.263+0000 7feefdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:40.268 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.268+0000 7feefdffb640 10 client.?.objecter ms_handle_connect 0x7fef081430c0 2024-03-25T19:45:40.268 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.268+0000 7feefdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:40.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.270+0000 7feefdffb640 1 -- 172.21.15.81:0/3955643339 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64202087 0 0) 0x7fef0806fa50 con 0x7fef0813f710 2024-03-25T19:45:40.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.270+0000 7feefdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.270+0000 7feefdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.270+0000 7feefdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.270+0000 7feefdffb640 10 monclient(hunting): my global_id is 4393 2024-03-25T19:45:40.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.270+0000 7feefdffb640 1 -- 172.21.15.81:0/3955643339 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeec004eb0 con 0x7fef0813f710 2024-03-25T19:45:40.275 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.276+0000 7feefdffb640 1 -- 172.21.15.81:0/3955643339 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 261333685 0 0) 0x7feeec004eb0 con 0x7fef0813f710 2024-03-25T19:45:40.275 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.276+0000 7feefdffb640 1 -- 172.21.15.81:0/3955643339 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef0806fa50 con 0x7fef0813f710 2024-03-25T19:45:40.276 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.276+0000 7feefdffb640 1 -- 172.21.15.81:0/3955643339 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7feef8001c80 con 0x7fef0813f710 2024-03-25T19:45:40.276 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.276+0000 7feefdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:40.276 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.276+0000 7feefdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:40.276 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.276+0000 7feefdffb640 10 monclient(hunting): dump: 2024-03-25T19:45:40.276 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:40.276 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:40.276 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:40.276 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:40.276 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:40.276 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:40.276 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:40.276 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:40.276 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:40.276 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:40.276 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.276+0000 7feefdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:40.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.282+0000 7feefdffb640 1 -- 172.21.15.81:0/3955643339 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428347523 0 0) 0x7fef080b73c0 con 0x7fef081430c0 2024-03-25T19:45:40.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.282+0000 7feefdffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.282+0000 7feefdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.282+0000 7feefdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.282+0000 7feefdffb640 10 monclient(hunting): my global_id is 4379 2024-03-25T19:45:40.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.282+0000 7feefdffb640 1 -- 172.21.15.81:0/3955643339 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeec004eb0 con 0x7fef081430c0 2024-03-25T19:45:40.286 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.286+0000 7feefdffb640 1 -- 172.21.15.81:0/3955643339 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 962031636 0 0) 0x7fef0806fa50 con 0x7fef0813f710 2024-03-25T19:45:40.286 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.286+0000 7feefdffb640 1 -- 172.21.15.81:0/3955643339 >> v1:172.21.15.81:6790/0 conn(0x7fef081430c0 legacy=0x7fef081d3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.286 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.286+0000 7feefdffb640 1 -- 172.21.15.81:0/3955643339 >> v1:172.21.15.81:6789/0 conn(0x7fef080a98f0 legacy=0x7fef08134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.286+0000 7feefdffb640 10 monclient: _finish_hunting 0 2024-03-25T19:45:40.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.286+0000 7feefdffb640 1 monclient: found mon.b 2024-03-25T19:45:40.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.286+0000 7feefdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:40.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.286+0000 7feefdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:40.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.286+0000 7feefdffb640 1 -- 172.21.15.81:0/3955643339 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef080b7550 con 0x7fef0813f710 2024-03-25T19:45:40.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.286+0000 7feefdffb640 10 monclient: _finish_auth 0 2024-03-25T19:45:40.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.286+0000 7feefdffb640 10 monclient: _check_auth_tickets 2024-03-25T19:45:40.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.286+0000 7feefdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:40.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.286+0000 7fef0f5cf640 5 monclient: authenticate success, global_id 4393 2024-03-25T19:45:40.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.286+0000 7fef0f5cf640 10 monclient: _renew_subs 2024-03-25T19:45:40.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.286+0000 7fef0f5cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:40.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.286+0000 7fef0f5cf640 1 -- 172.21.15.81:0/3955643339 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef0806e360 con 0x7fef0813f710 2024-03-25T19:45:40.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.287+0000 7fef0f5cf640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:40.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.287+0000 7fef0f5cf640 10 monclient: _renew_subs 2024-03-25T19:45:40.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.287+0000 7fef0f5cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:40.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.287+0000 7fef0f5cf640 1 -- 172.21.15.81:0/3955643339 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef08135880 con 0x7fef0813f710 2024-03-25T19:45:40.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.287+0000 7fef0f5cf640 1 librados: init done 2024-03-25T19:45:40.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.288+0000 7fef0f5cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:40.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.288+0000 7fef0f5cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:40.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.288+0000 7fef0f5cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:40.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.288+0000 7fef0f5cf640 1 -- 172.21.15.81:0/3955643339 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef0806f890 con 0x7fef0813f710 2024-03-25T19:45:40.550 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.551+0000 7feefdffb640 1 -- 172.21.15.81:0/3955643339 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feef8001210 con 0x7fef0813f710 2024-03-25T19:45:40.551 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.551+0000 7feefdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:40.551 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.551+0000 7feefdffb640 1 -- 172.21.15.81:0/3955643339 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7feef8003bc0 con 0x7fef0813f710 2024-03-25T19:45:40.551 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.551+0000 7feefdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:40.551 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.551+0000 7feefdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:40.551 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.551+0000 7feefdffb640 10 monclient: dump: 2024-03-25T19:45:40.551 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:40.551 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:40.551 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:40.551 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:40.551 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:40.551 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:40.551 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:40.551 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:40.551 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:40.551 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:40.552 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.551+0000 7feeff7fe640 4 set_mon_vals no callback set 2024-03-25T19:45:40.552 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.552+0000 7feefdffb640 1 -- 172.21.15.81:0/3955643339 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7feef8003de0 con 0x7fef0813f710 2024-03-25T19:45:40.552 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.553+0000 7fef0d36b640 10 monclient: get_auth_request con 0x7feeec06dd90 auth_method 0 2024-03-25T19:45:40.552 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.553+0000 7feefdffb640 10 client.4393.objecter ms_handle_connect 0x7feeec06dd90 2024-03-25T19:45:40.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.556+0000 7feefdffb640 1 -- 172.21.15.81:0/3955643339 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7feef8054970 con 0x7fef0813f710 2024-03-25T19:45:40.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.556+0000 7feefdffb640 10 client.4393.objecter ms_dispatch 0x7fef08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:40.556 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.556+0000 7feefdffb640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:40.556 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.556+0000 7feefdffb640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:40.556 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.556+0000 7feefdffb640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-25T19:45:40.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.558+0000 7feefdffb640 1 -- 172.21.15.81:0/3955643339 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feef8050480 con 0x7fef0813f710 2024-03-25T19:45:40.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.558+0000 7feefdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:40.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.558+0000 7feefdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:40.778 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.779+0000 7fef0f5cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:40.779 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.779+0000 7fef0f5cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:40.779 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.779+0000 7fef0f5cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:45:40.779 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.779+0000 7fef0f5cf640 1 -- 172.21.15.81:0/3955643339 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef0806e4f0 con 0x7fef0813f710 2024-03-25T19:45:40.813 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.813+0000 7feefdffb640 1 -- 172.21.15.81:0/3955643339 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7feef80523b0 con 0x7fef0813f710 2024-03-25T19:45:40.813 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.813+0000 7feefdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:40.813 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.813+0000 7feefdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:40.813 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:45:40.813 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:40.815 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.815+0000 7fef0f5cf640 10 librados: watch_flush enter 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.815+0000 7fef0f5cf640 10 librados: watch_flush exit 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.815+0000 7fef0f5cf640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.815+0000 7fef0f5cf640 10 client.4393.objecter successfully canceled tick 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.815+0000 7fef0f5cf640 1 -- 172.21.15.81:0/3955643339 >> v1:172.21.15.98:6800/3980829775 conn(0x7feeec06dd90 legacy=0x7feeec070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.815+0000 7fef0f5cf640 10 monclient: shutdown 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.815+0000 7fef0f5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.815+0000 7fef0f5cf640 1 -- 172.21.15.81:0/3955643339 >> v1:172.21.15.98:6789/0 conn(0x7fef0813f710 legacy=0x7fef081d3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.816+0000 7fef0f5cf640 1 -- 172.21.15.81:0/3955643339 shutdown_connections 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.816+0000 7fef0f5cf640 1 -- 172.21.15.81:0/3955643339 >> 172.21.15.81:0/3955643339 conn(0x7fef080b7ea0 msgr2=0x7fef08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.816+0000 7fef0f5cf640 1 -- 172.21.15.81:0/3955643339 shutdown_connections 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.816+0000 7fef0f5cf640 1 -- 172.21.15.81:0/3955643339 wait complete. 2024-03-25T19:45:40.816 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:40.816+0000 7fef0f5cf640 1 librados: shutdown 2024-03-25T19:45:40.827 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:47.634 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:47.751 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.751+0000 7f50b766e640 10 monclient: get_monmap_and_config 2024-03-25T19:45:47.751 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.751+0000 7f50b766e640 10 monclient: build_initial_monmap 2024-03-25T19:45:47.751 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.751+0000 7f50b766e640 10 monclient: monmap: 2024-03-25T19:45:47.752 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:45:47.752 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:47.752 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:45:47.752870+0000 2024-03-25T19:45:47.752 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:45:47.752870+0000 2024-03-25T19:45:47.752 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:47.752 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:47.752 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:45:47.752 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:45:47.752 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:45:47.752 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:47.753 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.752+0000 7f50b766e640 1 Processor -- start 2024-03-25T19:45:47.753 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.752+0000 7f50b766e640 1 -- start start 2024-03-25T19:45:47.754 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.752+0000 7f50b766e640 10 monclient: init 2024-03-25T19:45:47.754 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.753+0000 7f50b766e640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:47.754 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.753+0000 7f50b766e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:47.754 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.753+0000 7f50b766e640 10 monclient(hunting): picked mon.noname-a con 0x7f50b00a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:45:47.754 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.753+0000 7f50b766e640 10 monclient(hunting): picked mon.noname-b con 0x7f50b013f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:45:47.754 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.753+0000 7f50b766e640 10 monclient(hunting): picked mon.noname-c con 0x7f50b01430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:45:47.754 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.753+0000 7f50b766e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b0142520 con 0x7f50b00a98f0 2024-03-25T19:45:47.754 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.753+0000 7f50b766e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b00b28a0 con 0x7f50b01430c0 2024-03-25T19:45:47.754 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.753+0000 7f50b766e640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b00b2a30 con 0x7f50b013f710 2024-03-25T19:45:47.754 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.753+0000 7f50b766e640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:47.754 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.753+0000 7f50b766e640 10 monclient(hunting): authenticate will time out at 4182.835718s 2024-03-25T19:45:47.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.764+0000 7f50b4c09640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f50b013f710 0x7f50b01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:39196/0 (socket says 172.21.15.81:39196) 2024-03-25T19:45:47.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.765+0000 7f50b4c09640 1 -- 172.21.15.81:0/3764224360 learned_addr learned my addr 172.21.15.81:0/3764224360 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:45:47.776 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.777+0000 7f50a7fff640 1 -- 172.21.15.81:0/3764224360 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579290029 0 0) 0x7f50b00b2a30 con 0x7f50b013f710 2024-03-25T19:45:47.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.777+0000 7f50a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:47.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.777+0000 7f50a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:47.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.777+0000 7f50a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:47.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.777+0000 7f50a7fff640 10 monclient(hunting): my global_id is 4382 2024-03-25T19:45:47.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.777+0000 7f50a7fff640 1 -- 172.21.15.81:0/3764224360 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f509c003520 con 0x7f50b013f710 2024-03-25T19:45:47.793 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.794+0000 7f50a7fff640 1 -- 172.21.15.81:0/3764224360 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 850958884 0 0) 0x7f50b00b28a0 con 0x7f50b01430c0 2024-03-25T19:45:47.793 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.794+0000 7f50a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:47.793 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.794+0000 7f50a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:47.794 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.794+0000 7f50a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:47.794 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.794+0000 7f50a7fff640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:45:47.794 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.794+0000 7f50a7fff640 1 -- 172.21.15.81:0/3764224360 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b00b2a30 con 0x7f50b01430c0 2024-03-25T19:45:47.798 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.798+0000 7f50a7fff640 1 -- 172.21.15.81:0/3764224360 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340063473 0 0) 0x7f50b0142520 con 0x7f50b00a98f0 2024-03-25T19:45:47.798 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.798+0000 7f50a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:47.798 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.798+0000 7f50a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:47.798 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.798+0000 7f50a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:47.798 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.798+0000 7f50a7fff640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:45:47.798 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.798+0000 7f50a7fff640 1 -- 172.21.15.81:0/3764224360 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b00b28a0 con 0x7f50b00a98f0 2024-03-25T19:45:47.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.800+0000 7f50a7fff640 1 -- 172.21.15.81:0/3764224360 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 932549631 0 0) 0x7f509c003520 con 0x7f50b013f710 2024-03-25T19:45:47.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.800+0000 7f50a7fff640 1 -- 172.21.15.81:0/3764224360 >> v1:172.21.15.98:6789/0 conn(0x7f50b01430c0 legacy=0x7f50b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:47.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.800+0000 7f50a7fff640 1 -- 172.21.15.81:0/3764224360 >> v1:172.21.15.81:6789/0 conn(0x7f50b00a98f0 legacy=0x7f50b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:47.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.800+0000 7f50a7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:45:47.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.800+0000 7f50a7fff640 1 monclient: found mon.noname-b 2024-03-25T19:45:47.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.800+0000 7f50a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:47.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.800+0000 7f50a7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:45:47.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.800+0000 7f50a7fff640 1 -- 172.21.15.81:0/3764224360 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b00b44c0 con 0x7f50b013f710 2024-03-25T19:45:47.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.800+0000 7f50a7fff640 10 monclient: _finish_auth 0 2024-03-25T19:45:47.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.800+0000 7f50a7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:45:47.800 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.800+0000 7f50a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:47.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.800+0000 7f50a7fff640 1 -- 172.21.15.81:0/3764224360 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f50a0001690 con 0x7f50b013f710 2024-03-25T19:45:47.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.800+0000 7f50a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:47.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.800+0000 7f50a7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:47.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.800+0000 7f50a7fff640 10 monclient: dump: 2024-03-25T19:45:47.801 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:47.801 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:47.801 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:47.801 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:47.801 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:47.801 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:47.801 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:47.801 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:47.801 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:47.801 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:47.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.800+0000 7f50b766e640 5 monclient: authenticate success, global_id 4382 2024-03-25T19:45:47.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.800+0000 7f50b766e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:47.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.819+0000 7f50a7fff640 1 -- 172.21.15.81:0/3764224360 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50a00017f0 con 0x7f50b013f710 2024-03-25T19:45:47.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.819+0000 7f50a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:47.819 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.819+0000 7f50a7fff640 1 -- 172.21.15.81:0/3764224360 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f50a0001c00 con 0x7f50b013f710 2024-03-25T19:45:47.819 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.819+0000 7f50a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:47.819 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.819+0000 7f50a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:47.819 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.819+0000 7f50a7fff640 10 monclient: dump: 2024-03-25T19:45:47.819 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:47.819 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:47.819 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:47.819 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:47.819 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:47.819 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:47.819 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:47.819 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:47.819 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:47.820 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:47.820 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.819+0000 7f50b766e640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:47.820 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.819+0000 7f50b766e640 4 set_mon_vals no callback set 2024-03-25T19:45:47.821 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.819+0000 7f50b766e640 10 monclient: shutdown 2024-03-25T19:45:47.822 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.819+0000 7f50b766e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:47.822 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.819+0000 7f50b766e640 1 -- 172.21.15.81:0/3764224360 >> v1:172.21.15.81:6790/0 conn(0x7f50b013f710 legacy=0x7f50b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:47.822 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.819+0000 7f50b766e640 1 -- 172.21.15.81:0/3764224360 shutdown_connections 2024-03-25T19:45:47.822 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.819+0000 7f50b766e640 1 -- 172.21.15.81:0/3764224360 >> 172.21.15.81:0/3764224360 conn(0x7f50b00b7ea0 msgr2=0x7f50b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:47.822 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.820+0000 7f50b766e640 1 -- 172.21.15.81:0/3764224360 shutdown_connections 2024-03-25T19:45:47.822 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.820+0000 7f50b766e640 1 -- 172.21.15.81:0/3764224360 wait complete. 2024-03-25T19:45:47.822 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.821+0000 7f50b766e640 10 monclient: build_initial_monmap 2024-03-25T19:45:47.822 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.821+0000 7f50b766e640 10 monclient: monmap: 2024-03-25T19:45:47.822 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:45:47.822 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:47.822 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:45:47.822 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:45:47.822 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:47.822 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:45:47.823 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:45:47.823 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:45:47.823 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:45:47.823 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:47.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.821+0000 7f50b766e640 1 librados: starting msgr at 2024-03-25T19:45:47.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.821+0000 7f50b766e640 1 librados: starting objecter 2024-03-25T19:45:47.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.821+0000 7f50b766e640 1 Processor -- start 2024-03-25T19:45:47.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.821+0000 7f50b766e640 1 -- start start 2024-03-25T19:45:47.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.821+0000 7f50b766e640 1 librados: setting wanted keys 2024-03-25T19:45:47.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.821+0000 7f50b766e640 1 librados: calling monclient init 2024-03-25T19:45:47.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.821+0000 7f50b766e640 10 monclient: init 2024-03-25T19:45:47.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.821+0000 7f50b766e640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:47.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.822+0000 7f50b766e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:47.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.822+0000 7f50b766e640 10 monclient(hunting): picked mon.noname-c con 0x7f50b00a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:45:47.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.822+0000 7f50b766e640 10 monclient(hunting): picked mon.noname-b con 0x7f50b013f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:45:47.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.822+0000 7f50b766e640 10 monclient(hunting): picked mon.noname-a con 0x7f50b01430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:45:47.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.822+0000 7f50b766e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b00ab8b0 con 0x7f50b01430c0 2024-03-25T19:45:47.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.822+0000 7f50b766e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b00abb90 con 0x7f50b013f710 2024-03-25T19:45:47.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.822+0000 7f50b766e640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b0137870 con 0x7f50b00a98f0 2024-03-25T19:45:47.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.822+0000 7f50b766e640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:47.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.822+0000 7f50b766e640 10 monclient(hunting): authenticate will time out at 4182.904056s 2024-03-25T19:45:47.835 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.836+0000 7f50b5c0b640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f50b01430c0 0x7f50b01e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:58020/0 (socket says 172.21.15.81:58020) 2024-03-25T19:45:47.835 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.836+0000 7f50b5c0b640 1 -- 172.21.15.81:0/2807513111 learned_addr learned my addr 172.21.15.81:0/2807513111 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:45:47.837 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.837+0000 7f50a5ffb640 10 client.?.objecter ms_handle_connect 0x7f50b00a98f0 2024-03-25T19:45:47.837 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.837+0000 7f50a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:47.837 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.838+0000 7f50a5ffb640 10 client.?.objecter ms_handle_connect 0x7f50b01430c0 2024-03-25T19:45:47.837 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.838+0000 7f50a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:47.843 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.843+0000 7f50a5ffb640 10 client.?.objecter ms_handle_connect 0x7f50b013f710 2024-03-25T19:45:47.843 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.843+0000 7f50a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:47.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.848+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876375549 0 0) 0x7f50b00ab8b0 con 0x7f50b01430c0 2024-03-25T19:45:47.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.848+0000 7f50a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:47.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.848+0000 7f50a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:47.848 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.848+0000 7f50a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:47.848 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.848+0000 7f50a5ffb640 10 monclient(hunting): my global_id is 4401 2024-03-25T19:45:47.848 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.848+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5094004eb0 con 0x7f50b01430c0 2024-03-25T19:45:47.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.849+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900232203 0 0) 0x7f50b0137870 con 0x7f50b00a98f0 2024-03-25T19:45:47.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.849+0000 7f50a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:47.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.849+0000 7f50a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:47.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.849+0000 7f50a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:47.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.849+0000 7f50a5ffb640 10 monclient(hunting): my global_id is 4385 2024-03-25T19:45:47.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.849+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b00ab8b0 con 0x7f50b00a98f0 2024-03-25T19:45:47.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.850+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594223610 0 0) 0x7f50b00abb90 con 0x7f50b013f710 2024-03-25T19:45:47.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.850+0000 7f50a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:47.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.850+0000 7f50a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:47.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.850+0000 7f50a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:47.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.850+0000 7f50a5ffb640 10 monclient(hunting): my global_id is 4399 2024-03-25T19:45:47.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.850+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b0137870 con 0x7f50b013f710 2024-03-25T19:45:47.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.864+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2903398606 0 0) 0x7f5094004eb0 con 0x7f50b01430c0 2024-03-25T19:45:47.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.864+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50b00abb90 con 0x7f50b01430c0 2024-03-25T19:45:47.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.864+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f50a8001c90 con 0x7f50b01430c0 2024-03-25T19:45:47.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.864+0000 7f50a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:47.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.864+0000 7f50a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:47.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.864+0000 7f50a5ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:47.864 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:47.864 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:47.864 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:47.864 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:47.864 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:47.864 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:47.865 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:47.865 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:47.865 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:47.865 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:47.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.864+0000 7f50a5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:47.869 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.870+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3379688967 0 0) 0x7f50b00ab8b0 con 0x7f50b00a98f0 2024-03-25T19:45:47.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.870+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5094004eb0 con 0x7f50b00a98f0 2024-03-25T19:45:47.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.870+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f50ac001870 con 0x7f50b00a98f0 2024-03-25T19:45:47.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.870+0000 7f50a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:47.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.870+0000 7f50a5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:47.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.870+0000 7f50a5ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:47.870 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:47.870 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:47.870 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:47.870 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:47.870 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:47.870 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:47.870 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:47.870 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:47.870 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:47.871 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:47.871 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.871+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4001985685 0 0) 0x7f50b0137870 con 0x7f50b013f710 2024-03-25T19:45:47.871 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.871+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50b00ab8b0 con 0x7f50b013f710 2024-03-25T19:45:47.871 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.871+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f50a0001b00 con 0x7f50b013f710 2024-03-25T19:45:47.871 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.871+0000 7f50a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:47.871 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.871+0000 7f50a5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:47.871 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.871+0000 7f50a5ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:47.872 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:47.872 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:47.872 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:47.872 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:47.872 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:47.872 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:47.872 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:47.872 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:47.872 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:47.872 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:47.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 200894440 0 0) 0x7f50b00abb90 con 0x7f50b01430c0 2024-03-25T19:45:47.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 >> v1:172.21.15.81:6790/0 conn(0x7f50b00a98f0 legacy=0x7f50b01350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:47.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 >> v1:172.21.15.98:6789/0 conn(0x7f50b013f710 legacy=0x7f50b01df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:47.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50a5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:45:47.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50a5ffb640 1 monclient: found mon.a 2024-03-25T19:45:47.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:47.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50a5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:45:47.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b01389e0 con 0x7f50b01430c0 2024-03-25T19:45:47.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50a5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:45:47.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50a5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:45:47.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:47.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50b766e640 5 monclient: authenticate success, global_id 4401 2024-03-25T19:45:47.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50b766e640 10 monclient: _renew_subs 2024-03-25T19:45:47.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50b766e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:45:47.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50b766e640 1 -- 172.21.15.81:0/2807513111 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b0136d90 con 0x7f50b01430c0 2024-03-25T19:45:47.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50b766e640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:47.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50b766e640 10 monclient: _renew_subs 2024-03-25T19:45:47.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50b766e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:45:47.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50b766e640 1 -- 172.21.15.81:0/2807513111 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b01384c0 con 0x7f50b01430c0 2024-03-25T19:45:47.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.874+0000 7f50b766e640 1 librados: init done 2024-03-25T19:45:47.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.875+0000 7f50b766e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:47.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.875+0000 7f50b766e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:47.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.875+0000 7f50b766e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:45:47.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.875+0000 7f50b766e640 1 -- 172.21.15.81:0/2807513111 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b0136f80 con 0x7f50b01430c0 2024-03-25T19:45:47.895 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.895+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50a8003970 con 0x7f50b01430c0 2024-03-25T19:45:47.895 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.895+0000 7f50a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:47.895 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.895+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f50a8003d80 con 0x7f50b01430c0 2024-03-25T19:45:47.895 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.895+0000 7f50a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:47.895 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.895+0000 7f50a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:47.895 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.895+0000 7f50a5ffb640 10 monclient: dump: 2024-03-25T19:45:47.896 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:47.896 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:47.896 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:47.896 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:47.896 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:47.896 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:47.896 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:47.896 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:47.896 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:47.896 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:47.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.895+0000 7f50a77fe640 4 set_mon_vals no callback set 2024-03-25T19:45:47.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.896+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f50a8003fa0 con 0x7f50b01430c0 2024-03-25T19:45:47.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.897+0000 7f50b540a640 10 monclient: get_auth_request con 0x7f509406e110 auth_method 0 2024-03-25T19:45:47.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.897+0000 7f50a5ffb640 10 client.4401.objecter ms_handle_connect 0x7f509406e110 2024-03-25T19:45:47.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.899+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f50a8054a40 con 0x7f50b01430c0 2024-03-25T19:45:47.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.899+0000 7f50a5ffb640 10 client.4401.objecter ms_dispatch 0x7f50b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:47.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.899+0000 7f50a5ffb640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:47.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.899+0000 7f50a5ffb640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:47.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:47.900+0000 7f50a5ffb640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-25T19:45:48.178 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.178+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f50a8050550 con 0x7f50b01430c0 2024-03-25T19:45:48.178 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.179+0000 7f50a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:48.178 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.179+0000 7f50a5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:48.432 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.432+0000 7f50b766e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:48.432 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.432+0000 7f50b766e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:48.432 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.432+0000 7f50b766e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:45:48.432 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.432+0000 7f50b766e640 1 -- 172.21.15.81:0/2807513111 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50b00627b0 con 0x7f50b01430c0 2024-03-25T19:45:48.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.454+0000 7f50a5ffb640 1 -- 172.21.15.81:0/2807513111 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f50a8052480 con 0x7f50b01430c0 2024-03-25T19:45:48.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.454+0000 7f50a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:48.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.454+0000 7f50a5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:48.454 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:45:48.454 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:48.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.457+0000 7f50b766e640 10 librados: watch_flush enter 2024-03-25T19:45:48.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.457+0000 7f50b766e640 10 librados: watch_flush exit 2024-03-25T19:45:48.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.457+0000 7f50b766e640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-25T19:45:48.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.457+0000 7f50b766e640 10 client.4401.objecter successfully canceled tick 2024-03-25T19:45:48.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.457+0000 7f50b766e640 1 -- 172.21.15.81:0/2807513111 >> v1:172.21.15.98:6800/3980829775 conn(0x7f509406e110 legacy=0x7f50940705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:48.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.457+0000 7f50b766e640 10 monclient: shutdown 2024-03-25T19:45:48.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.457+0000 7f50b766e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:48.458 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.457+0000 7f50b766e640 1 -- 172.21.15.81:0/2807513111 >> v1:172.21.15.81:6789/0 conn(0x7f50b01430c0 legacy=0x7f50b01e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:48.458 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.457+0000 7f50b766e640 1 -- 172.21.15.81:0/2807513111 shutdown_connections 2024-03-25T19:45:48.458 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.457+0000 7f50b766e640 1 -- 172.21.15.81:0/2807513111 >> 172.21.15.81:0/2807513111 conn(0x7f50b00b7ea0 msgr2=0x7f50b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:48.458 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.457+0000 7f50b766e640 1 -- 172.21.15.81:0/2807513111 shutdown_connections 2024-03-25T19:45:48.458 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.457+0000 7f50b766e640 1 -- 172.21.15.81:0/2807513111 wait complete. 2024-03-25T19:45:48.458 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:48.457+0000 7f50b766e640 1 librados: shutdown 2024-03-25T19:45:48.468 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:55.274 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:55.390 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.390+0000 7efea30cf640 10 monclient: get_monmap_and_config 2024-03-25T19:45:55.390 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.390+0000 7efea30cf640 10 monclient: build_initial_monmap 2024-03-25T19:45:55.390 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.390+0000 7efea30cf640 10 monclient: monmap: 2024-03-25T19:45:55.390 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:45:55.390 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:55.390 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:45:55.391857+0000 2024-03-25T19:45:55.391 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:45:55.391857+0000 2024-03-25T19:45:55.391 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:55.391 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:55.391 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:45:55.391 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:45:55.391 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:45:55.391 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:55.392 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.391+0000 7efea30cf640 1 Processor -- start 2024-03-25T19:45:55.392 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.392+0000 7efea30cf640 1 -- start start 2024-03-25T19:45:55.392 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.392+0000 7efea30cf640 10 monclient: init 2024-03-25T19:45:55.392 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.392+0000 7efea30cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:55.392 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.392+0000 7efea30cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:55.392 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.392+0000 7efea30cf640 10 monclient(hunting): picked mon.noname-c con 0x7efe9c0b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:45:55.392 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.392+0000 7efea30cf640 10 monclient(hunting): picked mon.noname-a con 0x7efe9c0b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:45:55.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.392+0000 7efea30cf640 10 monclient(hunting): picked mon.noname-b con 0x7efe9c14be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:45:55.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.392+0000 7efea30cf640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c0af940 con 0x7efe9c0b1730 2024-03-25T19:45:55.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.392+0000 7efea30cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c0b0a70 con 0x7efe9c0b4d60 2024-03-25T19:45:55.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.392+0000 7efea30cf640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c0b0d50 con 0x7efe9c14be90 2024-03-25T19:45:55.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.392+0000 7efea30cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:55.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.392+0000 7efea30cf640 10 monclient(hunting): authenticate will time out at 4190.474724s 2024-03-25T19:45:55.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.405+0000 7efea0e6b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efe9c0b4d60 0x7efe9c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:48480/0 (socket says 172.21.15.81:48480) 2024-03-25T19:45:55.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.405+0000 7efea0e6b640 1 -- 172.21.15.81:0/1112771509 learned_addr learned my addr 172.21.15.81:0/1112771509 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:45:55.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.420+0000 7efe9b7fe640 1 -- 172.21.15.81:0/1112771509 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086269956 0 0) 0x7efe9c0af940 con 0x7efe9c0b1730 2024-03-25T19:45:55.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.420+0000 7efe9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:55.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.420+0000 7efe9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:55.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.420+0000 7efe9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:55.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.420+0000 7efe9b7fe640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:45:55.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.420+0000 7efe9b7fe640 1 -- 172.21.15.81:0/1112771509 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe8c003520 con 0x7efe9c0b1730 2024-03-25T19:45:55.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.424+0000 7efe9b7fe640 1 -- 172.21.15.81:0/1112771509 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792061391 0 0) 0x7efe9c0b0a70 con 0x7efe9c0b4d60 2024-03-25T19:45:55.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.424+0000 7efe9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:55.424 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.424+0000 7efe9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:55.424 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.424+0000 7efe9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:55.424 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.424+0000 7efe9b7fe640 10 monclient(hunting): my global_id is 4402 2024-03-25T19:45:55.424 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.424+0000 7efe9b7fe640 1 -- 172.21.15.81:0/1112771509 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe9c0af940 con 0x7efe9c0b4d60 2024-03-25T19:45:55.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.428+0000 7efe9b7fe640 1 -- 172.21.15.81:0/1112771509 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 851732428 0 0) 0x7efe8c003520 con 0x7efe9c0b1730 2024-03-25T19:45:55.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.428+0000 7efe9b7fe640 1 -- 172.21.15.81:0/1112771509 >> v1:172.21.15.81:6790/0 conn(0x7efe9c14be90 legacy=0x7efe9c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:55.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.428+0000 7efe9b7fe640 1 -- 172.21.15.81:0/1112771509 >> v1:172.21.15.98:6789/0 conn(0x7efe9c0b4d60 legacy=0x7efe9c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:55.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.428+0000 7efe9b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:55.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.428+0000 7efe9b7fe640 1 monclient: found mon.noname-a 2024-03-25T19:45:55.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.428+0000 7efe9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:55.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.428+0000 7efe9b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:45:55.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.428+0000 7efe9b7fe640 1 -- 172.21.15.81:0/1112771509 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c0b1030 con 0x7efe9c0b1730 2024-03-25T19:45:55.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.429+0000 7efe9b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:55.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.429+0000 7efe9b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:55.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.429+0000 7efe9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:55.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.429+0000 7efea30cf640 5 monclient: authenticate success, global_id 4404 2024-03-25T19:45:55.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.429+0000 7efea30cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:55.430 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.430+0000 7efe9b7fe640 1 -- 172.21.15.81:0/1112771509 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7efe88001650 con 0x7efe9c0b1730 2024-03-25T19:45:55.430 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.430+0000 7efe9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:55.430 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.430+0000 7efe9b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:55.430 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.430+0000 7efe9b7fe640 10 monclient: dump: 2024-03-25T19:45:55.430 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:55.430 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:55.430 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:55.431 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:55.431 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:55.431 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:55.431 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:55.431 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:55.431 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:55.431 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:55.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.430+0000 7efea30cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:55.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.435+0000 7efe9b7fe640 1 -- 172.21.15.81:0/1112771509 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe880018a0 con 0x7efe9c0b1730 2024-03-25T19:45:55.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.435+0000 7efe9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:55.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.435+0000 7efe9b7fe640 1 -- 172.21.15.81:0/1112771509 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7efe88001c30 con 0x7efe9c0b1730 2024-03-25T19:45:55.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.435+0000 7efe9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:55.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.435+0000 7efe9b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:55.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.435+0000 7efe9b7fe640 10 monclient: dump: 2024-03-25T19:45:55.435 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:55.435 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:55.435 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:55.435 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:55.435 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:55.435 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:55.436 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:55.436 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:55.436 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:55.436 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:55.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.435+0000 7efea30cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:55.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.435+0000 7efea30cf640 4 set_mon_vals no callback set 2024-03-25T19:45:55.437 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.436+0000 7efea30cf640 10 monclient: shutdown 2024-03-25T19:45:55.437 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.436+0000 7efea30cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:55.437 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.436+0000 7efea30cf640 1 -- 172.21.15.81:0/1112771509 >> v1:172.21.15.81:6789/0 conn(0x7efe9c0b1730 legacy=0x7efe9c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:55.438 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.436+0000 7efea30cf640 1 -- 172.21.15.81:0/1112771509 shutdown_connections 2024-03-25T19:45:55.438 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.436+0000 7efea30cf640 1 -- 172.21.15.81:0/1112771509 >> 172.21.15.81:0/1112771509 conn(0x7efe9c0617a0 msgr2=0x7efe9c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:55.438 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.436+0000 7efea30cf640 1 -- 172.21.15.81:0/1112771509 shutdown_connections 2024-03-25T19:45:55.438 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.436+0000 7efea30cf640 1 -- 172.21.15.81:0/1112771509 wait complete. 2024-03-25T19:45:55.438 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.437+0000 7efea30cf640 10 monclient: build_initial_monmap 2024-03-25T19:45:55.438 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.437+0000 7efea30cf640 10 monclient: monmap: 2024-03-25T19:45:55.438 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:45:55.438 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:55.438 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:45:55.438 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:45:55.438 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:55.438 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:45:55.438 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:45:55.439 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:45:55.439 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:45:55.439 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:55.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.437+0000 7efea30cf640 1 librados: starting msgr at 2024-03-25T19:45:55.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.437+0000 7efea30cf640 1 librados: starting objecter 2024-03-25T19:45:55.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.437+0000 7efea30cf640 1 Processor -- start 2024-03-25T19:45:55.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.437+0000 7efea30cf640 1 -- start start 2024-03-25T19:45:55.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.437+0000 7efea30cf640 1 librados: setting wanted keys 2024-03-25T19:45:55.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.437+0000 7efea30cf640 1 librados: calling monclient init 2024-03-25T19:45:55.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.437+0000 7efea30cf640 10 monclient: init 2024-03-25T19:45:55.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.438+0000 7efea30cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:55.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.438+0000 7efea30cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:55.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.438+0000 7efea30cf640 10 monclient(hunting): picked mon.noname-a con 0x7efe9c0b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:45:55.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.438+0000 7efea30cf640 10 monclient(hunting): picked mon.noname-c con 0x7efe9c0b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:45:55.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.438+0000 7efea30cf640 10 monclient(hunting): picked mon.noname-b con 0x7efe9c14be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:45:55.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.438+0000 7efea30cf640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c138650 con 0x7efe9c0b1730 2024-03-25T19:45:55.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.438+0000 7efea30cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c138930 con 0x7efe9c14be90 2024-03-25T19:45:55.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.438+0000 7efea30cf640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe9c136e70 con 0x7efe9c0b4d60 2024-03-25T19:45:55.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.438+0000 7efea30cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:55.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.438+0000 7efea30cf640 10 monclient(hunting): authenticate will time out at 4190.520721s 2024-03-25T19:45:55.444 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.444+0000 7efea0e6b640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7efe9c0b1730 0x7efe9c12fcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34848/0 (socket says 172.21.15.81:34848) 2024-03-25T19:45:55.444 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.444+0000 7efea0e6b640 1 -- 172.21.15.81:0/2311095433 learned_addr learned my addr 172.21.15.81:0/2311095433 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:45:55.446 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.447+0000 7efe997fa640 10 client.?.objecter ms_handle_connect 0x7efe9c0b1730 2024-03-25T19:45:55.446 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.447+0000 7efe997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:55.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.451+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663103541 0 0) 0x7efe9c138650 con 0x7efe9c0b1730 2024-03-25T19:45:55.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.451+0000 7efe997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:55.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.451+0000 7efe997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:55.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.451+0000 7efe997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:55.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.451+0000 7efe997fa640 10 monclient(hunting): my global_id is 4407 2024-03-25T19:45:55.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.451+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe80004eb0 con 0x7efe9c0b1730 2024-03-25T19:45:55.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.454+0000 7efe997fa640 10 client.?.objecter ms_handle_connect 0x7efe9c0b4d60 2024-03-25T19:45:55.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.454+0000 7efe997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:55.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.457+0000 7efe997fa640 10 client.?.objecter ms_handle_connect 0x7efe9c14be90 2024-03-25T19:45:55.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.457+0000 7efe997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:55.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.466+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337695610 0 0) 0x7efe9c138930 con 0x7efe9c14be90 2024-03-25T19:45:55.466 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.466+0000 7efe997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:55.466 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.466+0000 7efe997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:55.466 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.466+0000 7efe997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:55.466 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.466+0000 7efe997fa640 10 monclient(hunting): my global_id is 4405 2024-03-25T19:45:55.466 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.466+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe9c138650 con 0x7efe9c14be90 2024-03-25T19:45:55.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.468+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 511859653 0 0) 0x7efe80004eb0 con 0x7efe9c0b1730 2024-03-25T19:45:55.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.469+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe9c138930 con 0x7efe9c0b1730 2024-03-25T19:45:55.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.469+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7efe900019b0 con 0x7efe9c0b1730 2024-03-25T19:45:55.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.469+0000 7efe997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:55.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.469+0000 7efe997fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:55.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.469+0000 7efe997fa640 10 monclient(hunting): dump: 2024-03-25T19:45:55.469 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:55.469 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:55.469 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:55.469 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:55.469 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:55.469 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:55.469 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:55.469 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:55.469 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:55.469 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:55.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.469+0000 7efe997fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:55.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.474+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248142878 0 0) 0x7efe9c136e70 con 0x7efe9c0b4d60 2024-03-25T19:45:55.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.474+0000 7efe997fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:55.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.474+0000 7efe997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:55.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.474+0000 7efe997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:55.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.474+0000 7efe997fa640 10 monclient(hunting): my global_id is 4391 2024-03-25T19:45:55.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.474+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe80004eb0 con 0x7efe9c0b4d60 2024-03-25T19:45:55.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1589381485 0 0) 0x7efe9c138930 con 0x7efe9c0b1730 2024-03-25T19:45:55.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 >> v1:172.21.15.81:6790/0 conn(0x7efe9c0b4d60 legacy=0x7efe9c1313f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:55.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 >> v1:172.21.15.98:6789/0 conn(0x7efe9c14be90 legacy=0x7efe9c131b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:55.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efe997fa640 10 monclient: _finish_hunting 0 2024-03-25T19:45:55.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efe997fa640 1 monclient: found mon.a 2024-03-25T19:45:55.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efe997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:55.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efe997fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:45:55.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe9c137000 con 0x7efe9c0b1730 2024-03-25T19:45:55.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efe997fa640 10 monclient: _finish_auth 0 2024-03-25T19:45:55.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efe997fa640 10 monclient: _check_auth_tickets 2024-03-25T19:45:55.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efe997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:55.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efea30cf640 5 monclient: authenticate success, global_id 4407 2024-03-25T19:45:55.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efea30cf640 10 monclient: _renew_subs 2024-03-25T19:45:55.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efea30cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:45:55.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efea30cf640 1 -- 172.21.15.81:0/2311095433 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe9c130400 con 0x7efe9c0b1730 2024-03-25T19:45:55.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efea30cf640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:55.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efea30cf640 10 monclient: _renew_subs 2024-03-25T19:45:55.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efea30cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:45:55.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efea30cf640 1 -- 172.21.15.81:0/2311095433 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe9c1375c0 con 0x7efe9c0b1730 2024-03-25T19:45:55.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.479+0000 7efea30cf640 1 librados: init done 2024-03-25T19:45:55.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.480+0000 7efea30cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:55.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.480+0000 7efea30cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:55.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.480+0000 7efea30cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:45:55.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.480+0000 7efea30cf640 1 -- 172.21.15.81:0/2311095433 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9c137dc0 con 0x7efe9c0b1730 2024-03-25T19:45:55.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.482+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe900047e0 con 0x7efe9c0b1730 2024-03-25T19:45:55.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.482+0000 7efe997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:55.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.482+0000 7efe9affd640 4 set_mon_vals no callback set 2024-03-25T19:45:55.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.485+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7efe90004bb0 con 0x7efe9c0b1730 2024-03-25T19:45:55.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.485+0000 7efe997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:55.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.485+0000 7efe997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:55.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.485+0000 7efe997fa640 10 monclient: dump: 2024-03-25T19:45:55.485 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:45:55.485 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:45:55.485 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:55.485 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:45:55.485 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:55.485 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:45:55.485 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:45:55.486 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:45:55.486 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:45:55.486 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:45:55.778 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.778+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7efe90004d90 con 0x7efe9c0b1730 2024-03-25T19:45:55.778 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.779+0000 7efe9bfff640 10 monclient: get_auth_request con 0x7efe8006de00 auth_method 0 2024-03-25T19:45:55.778 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.779+0000 7efe997fa640 10 client.4407.objecter ms_handle_connect 0x7efe8006de00 2024-03-25T19:45:55.784 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.784+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7efe90054950 con 0x7efe9c0b1730 2024-03-25T19:45:55.784 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.784+0000 7efe997fa640 10 client.4407.objecter ms_dispatch 0x7efe9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:55.784 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.784+0000 7efe997fa640 3 client.4407.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:55.784 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.784+0000 7efe997fa640 3 client.4407.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:55.784 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.784+0000 7efe997fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-03-25T19:45:55.784 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.784+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe90050460 con 0x7efe9c0b1730 2024-03-25T19:45:55.784 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.784+0000 7efe997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:55.784 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.784+0000 7efe997fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:55.999 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.999+0000 7efea30cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:55.999 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.999+0000 7efea30cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:55.999 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.999+0000 7efea30cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:45:55.999 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:55.999+0000 7efea30cf640 1 -- 172.21.15.81:0/2311095433 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe9c137190 con 0x7efe9c0b1730 2024-03-25T19:45:56.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:56.072+0000 7efe997fa640 1 -- 172.21.15.81:0/2311095433 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7efe9c137190 con 0x7efe9c0b1730 2024-03-25T19:45:56.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:56.072+0000 7efe997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:56.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:56.072+0000 7efe997fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:56.072 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:45:56.072 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:56.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:56.074+0000 7efea30cf640 10 librados: watch_flush enter 2024-03-25T19:45:56.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:56.074+0000 7efea30cf640 10 librados: watch_flush exit 2024-03-25T19:45:56.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:56.074+0000 7efea30cf640 20 client.4407.objecter shutdown clearing up homeless session... 2024-03-25T19:45:56.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:56.074+0000 7efea30cf640 10 client.4407.objecter successfully canceled tick 2024-03-25T19:45:56.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:56.074+0000 7efea30cf640 1 -- 172.21.15.81:0/2311095433 >> v1:172.21.15.98:6800/3980829775 conn(0x7efe8006de00 legacy=0x7efe800702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:56.074+0000 7efea30cf640 10 monclient: shutdown 2024-03-25T19:45:56.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:56.074+0000 7efea30cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:56.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:56.074+0000 7efea30cf640 1 -- 172.21.15.81:0/2311095433 >> v1:172.21.15.81:6789/0 conn(0x7efe9c0b1730 legacy=0x7efe9c12fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:56.075+0000 7efea30cf640 1 -- 172.21.15.81:0/2311095433 shutdown_connections 2024-03-25T19:45:56.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:56.075+0000 7efea30cf640 1 -- 172.21.15.81:0/2311095433 >> 172.21.15.81:0/2311095433 conn(0x7efe9c0617a0 msgr2=0x7efe9c0b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:56.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:56.075+0000 7efea30cf640 1 -- 172.21.15.81:0/2311095433 shutdown_connections 2024-03-25T19:45:56.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:56.075+0000 7efea30cf640 1 -- 172.21.15.81:0/2311095433 wait complete. 2024-03-25T19:45:56.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:45:56.075+0000 7efea30cf640 1 librados: shutdown 2024-03-25T19:45:56.086 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:02.894 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:03.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.010+0000 7f65352c9640 10 monclient: get_monmap_and_config 2024-03-25T19:46:03.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.010+0000 7f65352c9640 10 monclient: build_initial_monmap 2024-03-25T19:46:03.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.010+0000 7f65352c9640 10 monclient: monmap: 2024-03-25T19:46:03.011 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:46:03.011 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:03.011 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:46:03.012352+0000 2024-03-25T19:46:03.011 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:46:03.012352+0000 2024-03-25T19:46:03.011 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:03.011 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:03.012 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:46:03.012 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:46:03.012 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:46:03.012 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:03.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.011+0000 7f65352c9640 1 Processor -- start 2024-03-25T19:46:03.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.011+0000 7f65352c9640 1 -- start start 2024-03-25T19:46:03.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.011+0000 7f65352c9640 10 monclient: init 2024-03-25T19:46:03.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.012+0000 7f65352c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:03.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.012+0000 7f65352c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:03.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.012+0000 7f65352c9640 10 monclient(hunting): picked mon.noname-a con 0x7f65300a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:46:03.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.012+0000 7f65352c9640 10 monclient(hunting): picked mon.noname-b con 0x7f653013f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:46:03.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.012+0000 7f65352c9640 10 monclient(hunting): picked mon.noname-c con 0x7f65301430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:46:03.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.012+0000 7f65352c9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6530142520 con 0x7f65300a98f0 2024-03-25T19:46:03.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.012+0000 7f65352c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65300b28a0 con 0x7f65301430c0 2024-03-25T19:46:03.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.012+0000 7f65352c9640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65300b2a30 con 0x7f653013f710 2024-03-25T19:46:03.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.012+0000 7f65352c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:03.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.012+0000 7f65352c9640 10 monclient(hunting): authenticate will time out at 4198.095122s 2024-03-25T19:46:03.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.023+0000 7f652e7fc640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f653013f710 0x7f65301429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:51646/0 (socket says 172.21.15.81:51646) 2024-03-25T19:46:03.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.023+0000 7f652e7fc640 1 -- 172.21.15.81:0/254072352 learned_addr learned my addr 172.21.15.81:0/254072352 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:46:03.041 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.041+0000 7f652dffb640 1 -- 172.21.15.81:0/254072352 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423318951 0 0) 0x7f65300b2a30 con 0x7f653013f710 2024-03-25T19:46:03.041 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.041+0000 7f652dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:03.041 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.041+0000 7f652dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:03.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.041+0000 7f652dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:03.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.041+0000 7f652dffb640 10 monclient(hunting): my global_id is 4394 2024-03-25T19:46:03.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.041+0000 7f652dffb640 1 -- 172.21.15.81:0/254072352 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6518003520 con 0x7f653013f710 2024-03-25T19:46:03.047 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.047+0000 7f652dffb640 1 -- 172.21.15.81:0/254072352 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694295787 0 0) 0x7f6530142520 con 0x7f65300a98f0 2024-03-25T19:46:03.047 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.047+0000 7f652dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:03.047 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.047+0000 7f652dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:03.048 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.047+0000 7f652dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:03.048 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.047+0000 7f652dffb640 10 monclient(hunting): my global_id is 4410 2024-03-25T19:46:03.048 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.047+0000 7f652dffb640 1 -- 172.21.15.81:0/254072352 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65300b2a30 con 0x7f65300a98f0 2024-03-25T19:46:03.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.051+0000 7f652dffb640 1 -- 172.21.15.81:0/254072352 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3326547537 0 0) 0x7f65300b28a0 con 0x7f65301430c0 2024-03-25T19:46:03.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.051+0000 7f652dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:03.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.051+0000 7f652dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:03.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.051+0000 7f652dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:03.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.051+0000 7f652dffb640 10 monclient(hunting): my global_id is 4408 2024-03-25T19:46:03.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.051+0000 7f652dffb640 1 -- 172.21.15.81:0/254072352 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6530142520 con 0x7f65301430c0 2024-03-25T19:46:03.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.072+0000 7f652dffb640 1 -- 172.21.15.81:0/254072352 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3997966816 0 0) 0x7f6518003520 con 0x7f653013f710 2024-03-25T19:46:03.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.072+0000 7f652dffb640 1 -- 172.21.15.81:0/254072352 >> v1:172.21.15.98:6789/0 conn(0x7f65301430c0 legacy=0x7f653014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.072+0000 7f652dffb640 1 -- 172.21.15.81:0/254072352 >> v1:172.21.15.81:6789/0 conn(0x7f65300a98f0 legacy=0x7f65300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.072+0000 7f652dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:46:03.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.072+0000 7f652dffb640 1 monclient: found mon.noname-b 2024-03-25T19:46:03.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.072+0000 7f652dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:03.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.072+0000 7f652dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:46:03.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.072+0000 7f652dffb640 1 -- 172.21.15.81:0/254072352 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65300b44c0 con 0x7f653013f710 2024-03-25T19:46:03.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.072+0000 7f652dffb640 10 monclient: _finish_auth 0 2024-03-25T19:46:03.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.072+0000 7f652dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:46:03.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.072+0000 7f652dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:03.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.073+0000 7f652dffb640 1 -- 172.21.15.81:0/254072352 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f6524001690 con 0x7f653013f710 2024-03-25T19:46:03.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.073+0000 7f652dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:03.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.073+0000 7f652dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:03.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.073+0000 7f652dffb640 10 monclient: dump: 2024-03-25T19:46:03.075 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:03.075 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:03.076 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:03.076 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:03.076 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:03.076 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:03.076 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:03.076 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:03.077 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:03.077 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:03.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.073+0000 7f65352c9640 5 monclient: authenticate success, global_id 4394 2024-03-25T19:46:03.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.073+0000 7f65352c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:03.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.106+0000 7f652dffb640 1 -- 172.21.15.81:0/254072352 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65240017f0 con 0x7f653013f710 2024-03-25T19:46:03.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.106+0000 7f652dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:03.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.106+0000 7f652dffb640 1 -- 172.21.15.81:0/254072352 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f6524001c00 con 0x7f653013f710 2024-03-25T19:46:03.108 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.106+0000 7f65352c9640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:03.108 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.106+0000 7f65352c9640 4 set_mon_vals no callback set 2024-03-25T19:46:03.108 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.106+0000 7f652dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:03.108 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.106+0000 7f652dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:03.108 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.106+0000 7f652dffb640 10 monclient: dump: 2024-03-25T19:46:03.108 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:03.108 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:03.108 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:03.108 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:03.108 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:03.108 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:03.108 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:03.108 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:03.108 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:03.109 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:03.111 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.107+0000 7f65352c9640 10 monclient: shutdown 2024-03-25T19:46:03.111 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.107+0000 7f65352c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:03.111 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.107+0000 7f65352c9640 1 -- 172.21.15.81:0/254072352 >> v1:172.21.15.81:6790/0 conn(0x7f653013f710 legacy=0x7f65301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.111 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.107+0000 7f65352c9640 1 -- 172.21.15.81:0/254072352 shutdown_connections 2024-03-25T19:46:03.111 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.107+0000 7f65352c9640 1 -- 172.21.15.81:0/254072352 >> 172.21.15.81:0/254072352 conn(0x7f65300b7ea0 msgr2=0x7f65300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:03.111 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.108+0000 7f65352c9640 1 -- 172.21.15.81:0/254072352 shutdown_connections 2024-03-25T19:46:03.111 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.108+0000 7f65352c9640 1 -- 172.21.15.81:0/254072352 wait complete. 2024-03-25T19:46:03.111 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.109+0000 7f65352c9640 10 monclient: build_initial_monmap 2024-03-25T19:46:03.111 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.109+0000 7f65352c9640 10 monclient: monmap: 2024-03-25T19:46:03.111 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:46:03.112 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:03.112 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:46:03.112 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:46:03.112 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:03.112 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:46:03.112 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:46:03.112 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:46:03.112 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:46:03.112 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:03.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.109+0000 7f65352c9640 1 librados: starting msgr at 2024-03-25T19:46:03.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.109+0000 7f65352c9640 1 librados: starting objecter 2024-03-25T19:46:03.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.109+0000 7f65352c9640 1 Processor -- start 2024-03-25T19:46:03.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.109+0000 7f65352c9640 1 -- start start 2024-03-25T19:46:03.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.109+0000 7f65352c9640 1 librados: setting wanted keys 2024-03-25T19:46:03.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.109+0000 7f65352c9640 1 librados: calling monclient init 2024-03-25T19:46:03.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.109+0000 7f65352c9640 10 monclient: init 2024-03-25T19:46:03.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.110+0000 7f65352c9640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:03.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.110+0000 7f65352c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:03.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.110+0000 7f65352c9640 10 monclient(hunting): picked mon.noname-b con 0x7f65300a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:46:03.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.110+0000 7f65352c9640 10 monclient(hunting): picked mon.noname-c con 0x7f653013f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:46:03.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.110+0000 7f65352c9640 10 monclient(hunting): picked mon.noname-a con 0x7f65301430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:46:03.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.110+0000 7f65352c9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6530137df0 con 0x7f65301430c0 2024-03-25T19:46:03.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.110+0000 7f65352c9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65301337a0 con 0x7f65300a98f0 2024-03-25T19:46:03.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.110+0000 7f65352c9640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6530133930 con 0x7f653013f710 2024-03-25T19:46:03.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.110+0000 7f65352c9640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:03.114 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.110+0000 7f65352c9640 10 monclient(hunting): authenticate will time out at 4198.193156s 2024-03-25T19:46:03.117 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.117+0000 7f652f7fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f65301430c0 0x7f6530133090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34172/0 (socket says 172.21.15.81:34172) 2024-03-25T19:46:03.118 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.117+0000 7f652f7fe640 1 -- 172.21.15.81:0/3373569491 learned_addr learned my addr 172.21.15.81:0/3373569491 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:46:03.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.119+0000 7f65177fe640 10 client.?.objecter ms_handle_connect 0x7f65301430c0 2024-03-25T19:46:03.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.119+0000 7f65177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:03.120 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.119+0000 7f65177fe640 10 client.?.objecter ms_handle_connect 0x7f65300a98f0 2024-03-25T19:46:03.120 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.119+0000 7f65177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:03.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.131+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86437050 0 0) 0x7f6530137df0 con 0x7f65301430c0 2024-03-25T19:46:03.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.131+0000 7f65177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:03.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.131+0000 7f65177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:03.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.131+0000 7f65177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:03.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.131+0000 7f65177fe640 10 monclient(hunting): my global_id is 4413 2024-03-25T19:46:03.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.131+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f651c004eb0 con 0x7f65301430c0 2024-03-25T19:46:03.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.131+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743501586 0 0) 0x7f65301337a0 con 0x7f65300a98f0 2024-03-25T19:46:03.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.131+0000 7f65177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:03.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.131+0000 7f65177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:03.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.131+0000 7f65177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:03.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.131+0000 7f65177fe640 10 monclient(hunting): my global_id is 4411 2024-03-25T19:46:03.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.131+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6530137df0 con 0x7f65300a98f0 2024-03-25T19:46:03.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.133+0000 7f65177fe640 10 client.?.objecter ms_handle_connect 0x7f653013f710 2024-03-25T19:46:03.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.133+0000 7f65177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:03.143 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.142+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2759227070 0 0) 0x7f6530137df0 con 0x7f65300a98f0 2024-03-25T19:46:03.143 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.142+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65301337a0 con 0x7f65300a98f0 2024-03-25T19:46:03.143 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.142+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f6520001970 con 0x7f65300a98f0 2024-03-25T19:46:03.143 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.142+0000 7f65177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:03.143 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.142+0000 7f65177fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:03.144 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.142+0000 7f65177fe640 10 monclient(hunting): dump: 2024-03-25T19:46:03.144 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:03.144 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:03.145 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:03.145 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:03.145 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:03.145 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:03.145 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:03.145 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:03.145 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:03.145 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:03.145 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.142+0000 7f65177fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:03.148 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.148+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3290678582 0 0) 0x7f651c004eb0 con 0x7f65301430c0 2024-03-25T19:46:03.148 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.148+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6530137df0 con 0x7f65301430c0 2024-03-25T19:46:03.148 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.148+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f6528001c70 con 0x7f65301430c0 2024-03-25T19:46:03.149 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.148+0000 7f65177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:03.149 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.148+0000 7f65177fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:03.149 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.148+0000 7f65177fe640 10 monclient(hunting): dump: 2024-03-25T19:46:03.149 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:03.149 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:03.149 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:03.149 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:03.149 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:03.149 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:03.149 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:03.149 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:03.149 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:03.149 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:03.153 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1301676183 0 0) 0x7f65301337a0 con 0x7f65300a98f0 2024-03-25T19:46:03.154 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 >> v1:172.21.15.81:6790/0 conn(0x7f653013f710 legacy=0x7f65301376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.154 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 >> v1:172.21.15.81:6789/0 conn(0x7f65301430c0 legacy=0x7f6530133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.154 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65177fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:03.154 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65177fe640 1 monclient: found mon.b 2024-03-25T19:46:03.154 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:03.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65177fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:46:03.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65301367d0 con 0x7f65300a98f0 2024-03-25T19:46:03.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65177fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:03.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65177fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:03.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:03.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65352c9640 5 monclient: authenticate success, global_id 4411 2024-03-25T19:46:03.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65352c9640 10 monclient: _renew_subs 2024-03-25T19:46:03.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65352c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:46:03.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65352c9640 1 -- 172.21.15.81:0/3373569491 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65301369e0 con 0x7f65300a98f0 2024-03-25T19:46:03.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65352c9640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:03.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65352c9640 10 monclient: _renew_subs 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65352c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65352c9640 1 -- 172.21.15.81:0/3373569491 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f653013b640 con 0x7f65300a98f0 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.151+0000 7f65352c9640 1 librados: init done 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.152+0000 7f65352c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.152+0000 7f65352c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:03.158 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.152+0000 7f65352c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:46:03.158 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.152+0000 7f65352c9640 1 -- 172.21.15.81:0/3373569491 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65300627b0 con 0x7f65300a98f0 2024-03-25T19:46:03.164 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.163+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6520003610 con 0x7f65300a98f0 2024-03-25T19:46:03.164 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.163+0000 7f65177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:03.164 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.163+0000 7f652d7fa640 4 set_mon_vals no callback set 2024-03-25T19:46:03.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.428+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f6520003920 con 0x7f65300a98f0 2024-03-25T19:46:03.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.428+0000 7f65177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:03.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.428+0000 7f65177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:03.429 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.428+0000 7f65177fe640 10 monclient: dump: 2024-03-25T19:46:03.429 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:03.429 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:03.430 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:03.430 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:03.430 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:03.430 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:03.430 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:03.430 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:03.430 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:03.430 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:03.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.429+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f6520019930 con 0x7f65300a98f0 2024-03-25T19:46:03.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.430+0000 7f652e7fc640 10 monclient: get_auth_request con 0x7f651c06de40 auth_method 0 2024-03-25T19:46:03.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.430+0000 7f65177fe640 10 client.4411.objecter ms_handle_connect 0x7f651c06de40 2024-03-25T19:46:03.432 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.431+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f65200536b0 con 0x7f65300a98f0 2024-03-25T19:46:03.432 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.431+0000 7f65177fe640 10 client.4411.objecter ms_dispatch 0x7f6530000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:03.432 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.431+0000 7f65177fe640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:03.432 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.431+0000 7f65177fe640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:03.432 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.431+0000 7f65177fe640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-25T19:46:03.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.435+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f652004eff0 con 0x7f65300a98f0 2024-03-25T19:46:03.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.435+0000 7f65177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:03.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.435+0000 7f65177fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:03.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.665+0000 7f65352c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:03.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.665+0000 7f65352c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:03.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.665+0000 7f65352c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:46:03.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.665+0000 7f65352c9640 1 -- 172.21.15.81:0/3373569491 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6530074cc0 con 0x7f65300a98f0 2024-03-25T19:46:03.697 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.696+0000 7f65177fe640 1 -- 172.21.15.81:0/3373569491 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f65200510f0 con 0x7f65300a98f0 2024-03-25T19:46:03.697 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.697+0000 7f65177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:03.698 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.697+0000 7f65177fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:03.698 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:46:03.698 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:03.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.698+0000 7f65352c9640 10 librados: watch_flush enter 2024-03-25T19:46:03.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.699+0000 7f65352c9640 10 librados: watch_flush exit 2024-03-25T19:46:03.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.699+0000 7f65352c9640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-25T19:46:03.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.699+0000 7f65352c9640 10 client.4411.objecter successfully canceled tick 2024-03-25T19:46:03.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.699+0000 7f65352c9640 1 -- 172.21.15.81:0/3373569491 >> v1:172.21.15.98:6800/3980829775 conn(0x7f651c06de40 legacy=0x7f651c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.699+0000 7f65352c9640 10 monclient: shutdown 2024-03-25T19:46:03.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.699+0000 7f65352c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:03.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.699+0000 7f65352c9640 1 -- 172.21.15.81:0/3373569491 >> v1:172.21.15.98:6789/0 conn(0x7f65300a98f0 legacy=0x7f65301392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.699+0000 7f65352c9640 1 -- 172.21.15.81:0/3373569491 shutdown_connections 2024-03-25T19:46:03.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.699+0000 7f65352c9640 1 -- 172.21.15.81:0/3373569491 >> 172.21.15.81:0/3373569491 conn(0x7f65300b7ea0 msgr2=0x7f6530074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:03.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.699+0000 7f65352c9640 1 -- 172.21.15.81:0/3373569491 shutdown_connections 2024-03-25T19:46:03.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.699+0000 7f65352c9640 1 -- 172.21.15.81:0/3373569491 wait complete. 2024-03-25T19:46:03.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:03.699+0000 7f65352c9640 1 librados: shutdown 2024-03-25T19:46:03.710 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:10.517 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:10.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.627+0000 7f5045c2a640 10 monclient: get_monmap_and_config 2024-03-25T19:46:10.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.627+0000 7f5045c2a640 10 monclient: build_initial_monmap 2024-03-25T19:46:10.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.627+0000 7f5045c2a640 10 monclient: monmap: 2024-03-25T19:46:10.626 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:46:10.627 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:10.627 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:46:10.628019+0000 2024-03-25T19:46:10.627 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:46:10.628019+0000 2024-03-25T19:46:10.627 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:10.627 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:10.627 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:46:10.627 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:46:10.627 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:46:10.627 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:10.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.627+0000 7f5045c2a640 1 Processor -- start 2024-03-25T19:46:10.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.628+0000 7f5045c2a640 1 -- start start 2024-03-25T19:46:10.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.628+0000 7f5045c2a640 10 monclient: init 2024-03-25T19:46:10.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.628+0000 7f5045c2a640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:10.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.628+0000 7f5045c2a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:10.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.628+0000 7f5045c2a640 10 monclient(hunting): picked mon.noname-b con 0x7f50400a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:46:10.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.628+0000 7f5045c2a640 10 monclient(hunting): picked mon.noname-a con 0x7f504013f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:46:10.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.628+0000 7f5045c2a640 10 monclient(hunting): picked mon.noname-c con 0x7f50401430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:46:10.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.628+0000 7f5045c2a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5040142520 con 0x7f504013f710 2024-03-25T19:46:10.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.628+0000 7f5045c2a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50400b28a0 con 0x7f50401430c0 2024-03-25T19:46:10.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.628+0000 7f5045c2a640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50400b2a30 con 0x7f50400a98f0 2024-03-25T19:46:10.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.628+0000 7f5045c2a640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:10.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.628+0000 7f5045c2a640 10 monclient(hunting): authenticate will time out at 4205.710627s 2024-03-25T19:46:10.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.641+0000 7f503f7fe640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f50400a98f0 0x7f50400a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:49980/0 (socket says 172.21.15.81:49980) 2024-03-25T19:46:10.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.641+0000 7f503f7fe640 1 -- 172.21.15.81:0/3954243884 learned_addr learned my addr 172.21.15.81:0/3954243884 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:46:10.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.659+0000 7f503e7fc640 1 -- 172.21.15.81:0/3954243884 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156240679 0 0) 0x7f50400b2a30 con 0x7f50400a98f0 2024-03-25T19:46:10.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.659+0000 7f503e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:10.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.659+0000 7f503e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:10.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.659+0000 7f503e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:10.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.659+0000 7f503e7fc640 10 monclient(hunting): my global_id is 4400 2024-03-25T19:46:10.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.659+0000 7f503e7fc640 1 -- 172.21.15.81:0/3954243884 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5028003520 con 0x7f50400a98f0 2024-03-25T19:46:10.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.666+0000 7f503e7fc640 1 -- 172.21.15.81:0/3954243884 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592760200 0 0) 0x7f50400b28a0 con 0x7f50401430c0 2024-03-25T19:46:10.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.666+0000 7f503e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:10.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.666+0000 7f503e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:10.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.666+0000 7f503e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:10.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.666+0000 7f503e7fc640 10 monclient(hunting): my global_id is 4414 2024-03-25T19:46:10.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.666+0000 7f503e7fc640 1 -- 172.21.15.81:0/3954243884 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50400b2a30 con 0x7f50401430c0 2024-03-25T19:46:10.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.673+0000 7f503e7fc640 1 -- 172.21.15.81:0/3954243884 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2123213112 0 0) 0x7f50400b2a30 con 0x7f50401430c0 2024-03-25T19:46:10.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.673+0000 7f503e7fc640 1 -- 172.21.15.81:0/3954243884 >> v1:172.21.15.81:6790/0 conn(0x7f50400a98f0 legacy=0x7f50400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:10.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.673+0000 7f503e7fc640 1 -- 172.21.15.81:0/3954243884 >> v1:172.21.15.81:6789/0 conn(0x7f504013f710 legacy=0x7f50401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:10.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.674+0000 7f503e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:10.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.674+0000 7f503e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:46:10.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.674+0000 7f503e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:10.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.674+0000 7f503e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:46:10.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.674+0000 7f503e7fc640 1 -- 172.21.15.81:0/3954243884 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50400b44c0 con 0x7f50401430c0 2024-03-25T19:46:10.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.674+0000 7f503e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:10.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.674+0000 7f503e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:10.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.674+0000 7f503e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:10.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.674+0000 7f503e7fc640 1 -- 172.21.15.81:0/3954243884 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f50380019f0 con 0x7f50401430c0 2024-03-25T19:46:10.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.674+0000 7f5045c2a640 5 monclient: authenticate success, global_id 4414 2024-03-25T19:46:10.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.674+0000 7f5045c2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:10.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.674+0000 7f503e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:10.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.674+0000 7f503e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:10.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.674+0000 7f503e7fc640 10 monclient: dump: 2024-03-25T19:46:10.675 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:10.675 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:10.675 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:10.675 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:10.675 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:10.675 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:10.675 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:10.675 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:10.675 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:10.675 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:10.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.674+0000 7f5045c2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:10.677 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.678+0000 7f503e7fc640 1 -- 172.21.15.81:0/3954243884 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5038001b50 con 0x7f50401430c0 2024-03-25T19:46:10.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.678+0000 7f503e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:10.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.678+0000 7f5045c2a640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:10.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.678+0000 7f5045c2a640 4 set_mon_vals no callback set 2024-03-25T19:46:10.679 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.678+0000 7f503e7fc640 1 -- 172.21.15.81:0/3954243884 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f5038001f20 con 0x7f50401430c0 2024-03-25T19:46:10.679 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.679+0000 7f503e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:10.679 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.679+0000 7f503e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:10.679 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.679+0000 7f503e7fc640 10 monclient: dump: 2024-03-25T19:46:10.679 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:10.679 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:10.679 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:10.679 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:10.679 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:10.679 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:10.679 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:10.679 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:10.679 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:10.680 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:10.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.679+0000 7f5045c2a640 10 monclient: shutdown 2024-03-25T19:46:10.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.679+0000 7f5045c2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:10.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.679+0000 7f5045c2a640 1 -- 172.21.15.81:0/3954243884 >> v1:172.21.15.98:6789/0 conn(0x7f50401430c0 legacy=0x7f504014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:10.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.679+0000 7f5045c2a640 1 -- 172.21.15.81:0/3954243884 shutdown_connections 2024-03-25T19:46:10.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.679+0000 7f5045c2a640 1 -- 172.21.15.81:0/3954243884 >> 172.21.15.81:0/3954243884 conn(0x7f50400b7ea0 msgr2=0x7f50400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:10.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.679+0000 7f5045c2a640 1 -- 172.21.15.81:0/3954243884 shutdown_connections 2024-03-25T19:46:10.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.679+0000 7f5045c2a640 1 -- 172.21.15.81:0/3954243884 wait complete. 2024-03-25T19:46:10.681 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.680+0000 7f5045c2a640 10 monclient: build_initial_monmap 2024-03-25T19:46:10.682 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.680+0000 7f5045c2a640 10 monclient: monmap: 2024-03-25T19:46:10.682 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:46:10.682 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:10.682 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:46:10.682 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:46:10.682 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:10.682 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:46:10.682 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:46:10.682 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:46:10.682 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:46:10.682 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:10.682 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.680+0000 7f5045c2a640 1 librados: starting msgr at 2024-03-25T19:46:10.682 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.680+0000 7f5045c2a640 1 librados: starting objecter 2024-03-25T19:46:10.682 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.680+0000 7f5045c2a640 1 Processor -- start 2024-03-25T19:46:10.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.680+0000 7f5045c2a640 1 -- start start 2024-03-25T19:46:10.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.680+0000 7f5045c2a640 1 librados: setting wanted keys 2024-03-25T19:46:10.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.680+0000 7f5045c2a640 1 librados: calling monclient init 2024-03-25T19:46:10.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.680+0000 7f5045c2a640 10 monclient: init 2024-03-25T19:46:10.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.681+0000 7f5045c2a640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:10.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.681+0000 7f5045c2a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:10.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.681+0000 7f5045c2a640 10 monclient(hunting): picked mon.noname-a con 0x7f50400a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:46:10.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.681+0000 7f5045c2a640 10 monclient(hunting): picked mon.noname-c con 0x7f504013f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:46:10.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.681+0000 7f5045c2a640 10 monclient(hunting): picked mon.noname-b con 0x7f50401430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:46:10.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.681+0000 7f5045c2a640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50401389a0 con 0x7f50400a98f0 2024-03-25T19:46:10.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.681+0000 7f5045c2a640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5040139310 con 0x7f50401430c0 2024-03-25T19:46:10.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.681+0000 7f5045c2a640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50401395f0 con 0x7f504013f710 2024-03-25T19:46:10.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.681+0000 7f5045c2a640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:10.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.681+0000 7f5045c2a640 10 monclient(hunting): authenticate will time out at 4205.763657s 2024-03-25T19:46:10.690 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.691+0000 7f503effd640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f504013f710 0x7f5040133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:50008/0 (socket says 172.21.15.81:50008) 2024-03-25T19:46:10.690 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.691+0000 7f503effd640 1 -- 172.21.15.81:0/3360601599 learned_addr learned my addr 172.21.15.81:0/3360601599 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:46:10.692 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.693+0000 7f5027fff640 10 client.?.objecter ms_handle_connect 0x7f504013f710 2024-03-25T19:46:10.692 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.693+0000 7f5027fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:10.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.702+0000 7f5027fff640 10 client.?.objecter ms_handle_connect 0x7f50401430c0 2024-03-25T19:46:10.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.702+0000 7f5027fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:10.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.702+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605084622 0 0) 0x7f50401395f0 con 0x7f504013f710 2024-03-25T19:46:10.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.702+0000 7f5027fff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:10.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.702+0000 7f5027fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:10.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.702+0000 7f5027fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:10.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.702+0000 7f5027fff640 10 monclient(hunting): my global_id is 4403 2024-03-25T19:46:10.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.702+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f502c004eb0 con 0x7f504013f710 2024-03-25T19:46:10.708 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.708+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 666971012 0 0) 0x7f5040139310 con 0x7f50401430c0 2024-03-25T19:46:10.708 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.708+0000 7f5027fff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:10.708 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.708+0000 7f5027fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:10.708 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.708+0000 7f5027fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:10.708 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.708+0000 7f5027fff640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:46:10.708 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.708+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50401395f0 con 0x7f50401430c0 2024-03-25T19:46:10.708 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.708+0000 7f5027fff640 10 client.?.objecter ms_handle_connect 0x7f50400a98f0 2024-03-25T19:46:10.708 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.708+0000 7f5027fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:10.713 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.713+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1246110794 0 0) 0x7f502c004eb0 con 0x7f504013f710 2024-03-25T19:46:10.713 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.713+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5040139310 con 0x7f504013f710 2024-03-25T19:46:10.713 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.713+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f5034001a30 con 0x7f504013f710 2024-03-25T19:46:10.713 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.713+0000 7f5027fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:10.713 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.713+0000 7f5027fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:10.713 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.713+0000 7f5027fff640 10 monclient(hunting): dump: 2024-03-25T19:46:10.713 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:10.713 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:10.713 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:10.713 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:10.713 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:10.713 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:10.713 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:10.713 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.713+0000 7f5027fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.714+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957378250 0 0) 0x7f50401389a0 con 0x7f50400a98f0 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.714+0000 7f5027fff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.714+0000 7f5027fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.714+0000 7f5027fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.714+0000 7f5027fff640 10 monclient(hunting): my global_id is 4419 2024-03-25T19:46:10.715 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.714+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f502c004eb0 con 0x7f50400a98f0 2024-03-25T19:46:10.721 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1390878033 0 0) 0x7f5040139310 con 0x7f504013f710 2024-03-25T19:46:10.721 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 >> v1:172.21.15.98:6789/0 conn(0x7f50401430c0 legacy=0x7f50401e2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:10.721 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 >> v1:172.21.15.81:6789/0 conn(0x7f50400a98f0 legacy=0x7f50401da780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:10.721 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5027fff640 10 monclient: _finish_hunting 0 2024-03-25T19:46:10.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5027fff640 1 monclient: found mon.c 2024-03-25T19:46:10.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5027fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:10.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5027fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:46:10.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50401398d0 con 0x7f504013f710 2024-03-25T19:46:10.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5027fff640 10 monclient: _finish_auth 0 2024-03-25T19:46:10.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5027fff640 10 monclient: _check_auth_tickets 2024-03-25T19:46:10.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5027fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:10.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5045c2a640 5 monclient: authenticate success, global_id 4403 2024-03-25T19:46:10.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5045c2a640 10 monclient: _renew_subs 2024-03-25T19:46:10.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5045c2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:46:10.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5045c2a640 1 -- 172.21.15.81:0/3360601599 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5040138480 con 0x7f504013f710 2024-03-25T19:46:10.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5045c2a640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:10.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5045c2a640 10 monclient: _renew_subs 2024-03-25T19:46:10.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5045c2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:46:10.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5045c2a640 1 -- 172.21.15.81:0/3360601599 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50401380d0 con 0x7f504013f710 2024-03-25T19:46:10.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.721+0000 7f5045c2a640 1 librados: init done 2024-03-25T19:46:10.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.722+0000 7f5045c2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:10.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.722+0000 7f5045c2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:10.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.722+0000 7f5045c2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:46:10.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:10.722+0000 7f5045c2a640 1 -- 172.21.15.81:0/3360601599 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50400627b0 con 0x7f504013f710 2024-03-25T19:46:11.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.028+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5034001c70 con 0x7f504013f710 2024-03-25T19:46:11.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.028+0000 7f5027fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:11.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.028+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f5034003a60 con 0x7f504013f710 2024-03-25T19:46:11.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.028+0000 7f5027fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:11.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.028+0000 7f5027fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:11.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.028+0000 7f5027fff640 10 monclient: dump: 2024-03-25T19:46:11.028 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:11.028 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:11.028 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:11.028 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:11.028 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:11.028 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:11.028 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:11.028 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:11.028 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:11.028 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:11.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.028+0000 7f503dffb640 4 set_mon_vals no callback set 2024-03-25T19:46:11.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.029+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f5034003c40 con 0x7f504013f710 2024-03-25T19:46:11.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.029+0000 7f503f7fe640 10 monclient: get_auth_request con 0x7f502c06de00 auth_method 0 2024-03-25T19:46:11.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.030+0000 7f5027fff640 10 client.4403.objecter ms_handle_connect 0x7f502c06de00 2024-03-25T19:46:11.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.034+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f50340548b0 con 0x7f504013f710 2024-03-25T19:46:11.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.034+0000 7f5027fff640 10 client.4403.objecter ms_dispatch 0x7f5040000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:11.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.034+0000 7f5027fff640 3 client.4403.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:11.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.034+0000 7f5027fff640 3 client.4403.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:11.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.034+0000 7f5027fff640 20 client.4403.objecter dump_active .. 0 homeless 2024-03-25T19:46:11.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.034+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f50340503c0 con 0x7f504013f710 2024-03-25T19:46:11.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.034+0000 7f5027fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:11.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.034+0000 7f5027fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:11.291 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.291+0000 7f5045c2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:11.291 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.291+0000 7f5045c2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:11.291 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.291+0000 7f5045c2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:46:11.291 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.291+0000 7f5045c2a640 1 -- 172.21.15.81:0/3360601599 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5040138610 con 0x7f504013f710 2024-03-25T19:46:11.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.314+0000 7f5027fff640 1 -- 172.21.15.81:0/3360601599 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f50340522f0 con 0x7f504013f710 2024-03-25T19:46:11.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.314+0000 7f5027fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:11.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.314+0000 7f5027fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:11.314 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:46:11.314 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:11.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.316+0000 7f5045c2a640 10 librados: watch_flush enter 2024-03-25T19:46:11.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.316+0000 7f5045c2a640 10 librados: watch_flush exit 2024-03-25T19:46:11.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.316+0000 7f5045c2a640 20 client.4403.objecter shutdown clearing up homeless session... 2024-03-25T19:46:11.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.316+0000 7f5045c2a640 10 client.4403.objecter successfully canceled tick 2024-03-25T19:46:11.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.316+0000 7f5045c2a640 1 -- 172.21.15.81:0/3360601599 >> v1:172.21.15.98:6800/3980829775 conn(0x7f502c06de00 legacy=0x7f502c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:11.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.316+0000 7f5045c2a640 10 monclient: shutdown 2024-03-25T19:46:11.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.316+0000 7f5045c2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:11.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.316+0000 7f5045c2a640 1 -- 172.21.15.81:0/3360601599 >> v1:172.21.15.81:6790/0 conn(0x7f504013f710 legacy=0x7f5040133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:11.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.316+0000 7f5045c2a640 1 -- 172.21.15.81:0/3360601599 shutdown_connections 2024-03-25T19:46:11.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.316+0000 7f5045c2a640 1 -- 172.21.15.81:0/3360601599 >> 172.21.15.81:0/3360601599 conn(0x7f50400b7ea0 msgr2=0x7f5040074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:11.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.317+0000 7f5045c2a640 1 -- 172.21.15.81:0/3360601599 shutdown_connections 2024-03-25T19:46:11.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.317+0000 7f5045c2a640 1 -- 172.21.15.81:0/3360601599 wait complete. 2024-03-25T19:46:11.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:11.317+0000 7f5045c2a640 1 librados: shutdown 2024-03-25T19:46:11.328 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:17.313 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:04:39.955 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:17.322 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:04:39.964 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:17.352 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:04:39.994 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:17.369 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:04:40.004 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:17.370 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:04:40.012 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:17.376 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:04:40.011 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:17.378 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:04:40.020 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:17.416 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:04:40.058 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:17.419 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:04:40.054 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:18.134 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:18.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.250+0000 7fa1f07cf640 10 monclient: get_monmap_and_config 2024-03-25T19:46:18.250 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.250+0000 7fa1f07cf640 10 monclient: build_initial_monmap 2024-03-25T19:46:18.250 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.250+0000 7fa1f07cf640 10 monclient: monmap: 2024-03-25T19:46:18.250 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:46:18.250 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:18.250 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:46:18.250984+0000 2024-03-25T19:46:18.250 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:46:18.250984+0000 2024-03-25T19:46:18.250 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:18.250 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:18.250 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:46:18.250 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:46:18.250 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:46:18.250 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:18.251 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.250+0000 7fa1f07cf640 1 Processor -- start 2024-03-25T19:46:18.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.250+0000 7fa1f07cf640 1 -- start start 2024-03-25T19:46:18.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.250+0000 7fa1f07cf640 10 monclient: init 2024-03-25T19:46:18.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.251+0000 7fa1f07cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:18.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.251+0000 7fa1f07cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:18.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.251+0000 7fa1f07cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa1e80a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:46:18.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.251+0000 7fa1f07cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa1e813f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:46:18.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.251+0000 7fa1f07cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa1e81430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:46:18.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.251+0000 7fa1f07cf640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e8142520 con 0x7fa1e80a98f0 2024-03-25T19:46:18.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.251+0000 7fa1f07cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e80b28a0 con 0x7fa1e813f710 2024-03-25T19:46:18.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.251+0000 7fa1f07cf640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e80b2a30 con 0x7fa1e81430c0 2024-03-25T19:46:18.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.251+0000 7fa1f07cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:18.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.251+0000 7fa1f07cf640 10 monclient(hunting): authenticate will time out at 4213.333579s 2024-03-25T19:46:18.266 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.266+0000 7fa1eed6c640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fa1e81430c0 0x7fa1e814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:48154/0 (socket says 172.21.15.81:48154) 2024-03-25T19:46:18.266 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.266+0000 7fa1eed6c640 1 -- 172.21.15.81:0/3899340057 learned_addr learned my addr 172.21.15.81:0/3899340057 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:46:18.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.280+0000 7fa1ed569640 1 -- 172.21.15.81:0/3899340057 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157605240 0 0) 0x7fa1e80b28a0 con 0x7fa1e813f710 2024-03-25T19:46:18.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.280+0000 7fa1ed569640 10 monclient(hunting): _negotiate 2024-03-25T19:46:18.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.280+0000 7fa1ed569640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:18.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.280+0000 7fa1ed569640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:18.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.280+0000 7fa1ed569640 10 monclient(hunting): my global_id is 4420 2024-03-25T19:46:18.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.280+0000 7fa1ed569640 1 -- 172.21.15.81:0/3899340057 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d4003520 con 0x7fa1e813f710 2024-03-25T19:46:18.290 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.291+0000 7fa1ed569640 1 -- 172.21.15.81:0/3899340057 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073831533 0 0) 0x7fa1e80b2a30 con 0x7fa1e81430c0 2024-03-25T19:46:18.290 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.291+0000 7fa1ed569640 10 monclient(hunting): _negotiate 2024-03-25T19:46:18.290 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.291+0000 7fa1ed569640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:18.290 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.291+0000 7fa1ed569640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:18.290 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.291+0000 7fa1ed569640 10 monclient(hunting): my global_id is 4406 2024-03-25T19:46:18.290 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.291+0000 7fa1ed569640 1 -- 172.21.15.81:0/3899340057 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1e80b28a0 con 0x7fa1e81430c0 2024-03-25T19:46:18.292 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.292+0000 7fa1ed569640 1 -- 172.21.15.81:0/3899340057 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 648771766 0 0) 0x7fa1d4003520 con 0x7fa1e813f710 2024-03-25T19:46:18.292 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.292+0000 7fa1ed569640 1 -- 172.21.15.81:0/3899340057 >> v1:172.21.15.81:6790/0 conn(0x7fa1e81430c0 legacy=0x7fa1e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:18.292 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.292+0000 7fa1ed569640 1 -- 172.21.15.81:0/3899340057 >> v1:172.21.15.81:6789/0 conn(0x7fa1e80a98f0 legacy=0x7fa1e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:18.292 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.292+0000 7fa1ed569640 10 monclient: _finish_hunting 0 2024-03-25T19:46:18.292 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.292+0000 7fa1ed569640 1 monclient: found mon.noname-c 2024-03-25T19:46:18.292 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.292+0000 7fa1ed569640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:18.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.292+0000 7fa1ed569640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:46:18.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.292+0000 7fa1ed569640 1 -- 172.21.15.81:0/3899340057 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e80b44c0 con 0x7fa1e813f710 2024-03-25T19:46:18.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.292+0000 7fa1ed569640 10 monclient: _finish_auth 0 2024-03-25T19:46:18.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.292+0000 7fa1ed569640 10 monclient: _check_auth_tickets 2024-03-25T19:46:18.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.292+0000 7fa1ed569640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:18.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.292+0000 7fa1ed569640 1 -- 172.21.15.81:0/3899340057 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa1d8001690 con 0x7fa1e813f710 2024-03-25T19:46:18.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.292+0000 7fa1ed569640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:18.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.292+0000 7fa1ed569640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:18.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.292+0000 7fa1ed569640 10 monclient: dump: 2024-03-25T19:46:18.293 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:18.293 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:18.293 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:18.293 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:18.293 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:18.294 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:18.294 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:18.294 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:18.294 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:18.294 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:18.294 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.292+0000 7fa1f07cf640 5 monclient: authenticate success, global_id 4420 2024-03-25T19:46:18.294 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.292+0000 7fa1f07cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:18.302 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.303+0000 7fa1ed569640 1 -- 172.21.15.81:0/3899340057 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1d80017f0 con 0x7fa1e813f710 2024-03-25T19:46:18.302 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.303+0000 7fa1ed569640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:18.302 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.303+0000 7fa1ed569640 1 -- 172.21.15.81:0/3899340057 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa1d8001c00 con 0x7fa1e813f710 2024-03-25T19:46:18.302 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.303+0000 7fa1f07cf640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:18.302 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.303+0000 7fa1f07cf640 4 set_mon_vals no callback set 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.303+0000 7fa1ed569640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.303+0000 7fa1ed569640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.303+0000 7fa1ed569640 10 monclient: dump: 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:18.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.303+0000 7fa1f07cf640 10 monclient: shutdown 2024-03-25T19:46:18.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.303+0000 7fa1f07cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:18.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.303+0000 7fa1f07cf640 1 -- 172.21.15.81:0/3899340057 >> v1:172.21.15.98:6789/0 conn(0x7fa1e813f710 legacy=0x7fa1e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:18.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.304+0000 7fa1f07cf640 1 -- 172.21.15.81:0/3899340057 shutdown_connections 2024-03-25T19:46:18.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.304+0000 7fa1f07cf640 1 -- 172.21.15.81:0/3899340057 >> 172.21.15.81:0/3899340057 conn(0x7fa1e80b7ea0 msgr2=0x7fa1e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:18.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.304+0000 7fa1f07cf640 1 -- 172.21.15.81:0/3899340057 shutdown_connections 2024-03-25T19:46:18.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.304+0000 7fa1f07cf640 1 -- 172.21.15.81:0/3899340057 wait complete. 2024-03-25T19:46:18.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.305+0000 7fa1f07cf640 10 monclient: build_initial_monmap 2024-03-25T19:46:18.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.305+0000 7fa1f07cf640 10 monclient: monmap: 2024-03-25T19:46:18.305 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:46:18.305 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:18.306 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:46:18.306 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:46:18.306 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:18.306 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:46:18.306 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:46:18.306 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:46:18.306 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:46:18.306 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:18.306 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.305+0000 7fa1f07cf640 1 librados: starting msgr at 2024-03-25T19:46:18.306 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.305+0000 7fa1f07cf640 1 librados: starting objecter 2024-03-25T19:46:18.306 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.305+0000 7fa1f07cf640 1 Processor -- start 2024-03-25T19:46:18.306 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.305+0000 7fa1f07cf640 1 -- start start 2024-03-25T19:46:18.306 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.305+0000 7fa1f07cf640 1 librados: setting wanted keys 2024-03-25T19:46:18.306 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.305+0000 7fa1f07cf640 1 librados: calling monclient init 2024-03-25T19:46:18.306 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.305+0000 7fa1f07cf640 10 monclient: init 2024-03-25T19:46:18.307 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.306+0000 7fa1f07cf640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:18.307 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.306+0000 7fa1f07cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:18.307 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.306+0000 7fa1f07cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa1e80a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:46:18.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.306+0000 7fa1f07cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa1e813f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:46:18.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.306+0000 7fa1f07cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa1e81430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:46:18.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.306+0000 7fa1f07cf640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e8139070 con 0x7fa1e813f710 2024-03-25T19:46:18.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.306+0000 7fa1f07cf640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e8133710 con 0x7fa1e81430c0 2024-03-25T19:46:18.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.306+0000 7fa1f07cf640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e81338a0 con 0x7fa1e80a98f0 2024-03-25T19:46:18.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.306+0000 7fa1f07cf640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:18.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.306+0000 7fa1f07cf640 10 monclient(hunting): authenticate will time out at 4213.388390s 2024-03-25T19:46:18.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.318+0000 7fa1edd6a640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fa1e813f710 0x7fa1e81daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:45838/0 (socket says 172.21.15.81:45838) 2024-03-25T19:46:18.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.318+0000 7fa1edd6a640 1 -- 172.21.15.81:0/2351629771 learned_addr learned my addr 172.21.15.81:0/2351629771 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:46:18.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.320+0000 7fa1deffd640 10 client.?.objecter ms_handle_connect 0x7fa1e813f710 2024-03-25T19:46:18.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.320+0000 7fa1deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:18.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.330+0000 7fa1deffd640 10 client.?.objecter ms_handle_connect 0x7fa1e81430c0 2024-03-25T19:46:18.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.330+0000 7fa1deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:18.331 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.331+0000 7fa1deffd640 10 client.?.objecter ms_handle_connect 0x7fa1e80a98f0 2024-03-25T19:46:18.331 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.331+0000 7fa1deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:18.332 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.332+0000 7fa1deffd640 1 -- 172.21.15.81:0/2351629771 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280271730 0 0) 0x7fa1e8139070 con 0x7fa1e813f710 2024-03-25T19:46:18.332 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.332+0000 7fa1deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:18.332 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.332+0000 7fa1deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:18.332 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.332+0000 7fa1deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:18.332 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.332+0000 7fa1deffd640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:46:18.332 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.332+0000 7fa1deffd640 1 -- 172.21.15.81:0/2351629771 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1cc004eb0 con 0x7fa1e813f710 2024-03-25T19:46:18.339 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.340+0000 7fa1deffd640 1 -- 172.21.15.81:0/2351629771 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 991703657 0 0) 0x7fa1cc004eb0 con 0x7fa1e813f710 2024-03-25T19:46:18.339 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.340+0000 7fa1deffd640 1 -- 172.21.15.81:0/2351629771 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1e8139070 con 0x7fa1e813f710 2024-03-25T19:46:18.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.343+0000 7fa1deffd640 1 -- 172.21.15.81:0/2351629771 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa1d80035f0 con 0x7fa1e813f710 2024-03-25T19:46:18.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.343+0000 7fa1deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:18.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.343+0000 7fa1deffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:18.343 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.343+0000 7fa1deffd640 10 monclient(hunting): dump: 2024-03-25T19:46:18.343 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:18.343 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:18.343 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:18.343 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:18.343 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:18.343 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:18.343 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:18.343 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:18.343 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:18.343 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:18.343 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.343+0000 7fa1deffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:18.346 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.346+0000 7fa1deffd640 1 -- 172.21.15.81:0/2351629771 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336555215 0 0) 0x7fa1e81338a0 con 0x7fa1e80a98f0 2024-03-25T19:46:18.346 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.347+0000 7fa1deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:18.346 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.347+0000 7fa1deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:18.346 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.347+0000 7fa1deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:18.346 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.347+0000 7fa1deffd640 10 monclient(hunting): my global_id is 4409 2024-03-25T19:46:18.346 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.347+0000 7fa1deffd640 1 -- 172.21.15.81:0/2351629771 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1cc004eb0 con 0x7fa1e80a98f0 2024-03-25T19:46:18.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1deffd640 1 -- 172.21.15.81:0/2351629771 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4077503685 0 0) 0x7fa1e8139070 con 0x7fa1e813f710 2024-03-25T19:46:18.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1deffd640 1 -- 172.21.15.81:0/2351629771 >> v1:172.21.15.81:6790/0 conn(0x7fa1e80a98f0 legacy=0x7fa1e81378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:18.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1deffd640 1 -- 172.21.15.81:0/2351629771 >> v1:172.21.15.98:6789/0 conn(0x7fa1e81430c0 legacy=0x7fa1e8133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:18.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1deffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:18.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1deffd640 1 monclient: found mon.a 2024-03-25T19:46:18.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:18.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1deffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:18.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1deffd640 1 -- 172.21.15.81:0/2351629771 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e8133a30 con 0x7fa1e813f710 2024-03-25T19:46:18.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1deffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:18.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1deffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:18.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:18.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1f07cf640 5 monclient: authenticate success, global_id 4425 2024-03-25T19:46:18.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1f07cf640 10 monclient: _renew_subs 2024-03-25T19:46:18.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1f07cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:18.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1f07cf640 1 -- 172.21.15.81:0/2351629771 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e80b7100 con 0x7fa1e813f710 2024-03-25T19:46:18.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1f07cf640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:18.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1f07cf640 10 monclient: _renew_subs 2024-03-25T19:46:18.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1f07cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:18.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1f07cf640 1 -- 172.21.15.81:0/2351629771 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e8136e60 con 0x7fa1e813f710 2024-03-25T19:46:18.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.350+0000 7fa1f07cf640 1 librados: init done 2024-03-25T19:46:18.352 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.351+0000 7fa1f07cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:18.352 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.351+0000 7fa1f07cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:18.352 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.351+0000 7fa1f07cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:18.352 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.351+0000 7fa1f07cf640 1 -- 172.21.15.81:0/2351629771 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e80627b0 con 0x7fa1e813f710 2024-03-25T19:46:18.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.650+0000 7fa1deffd640 1 -- 172.21.15.81:0/2351629771 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1d8003b40 con 0x7fa1e813f710 2024-03-25T19:46:18.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.650+0000 7fa1deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:18.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.650+0000 7fa1deffd640 1 -- 172.21.15.81:0/2351629771 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa1d8003f10 con 0x7fa1e813f710 2024-03-25T19:46:18.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.650+0000 7fa1deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:18.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.650+0000 7fa1deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:18.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.650+0000 7fa1deffd640 10 monclient: dump: 2024-03-25T19:46:18.650 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:18.651 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:18.651 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:18.651 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:18.651 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:18.651 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:18.651 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:18.651 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:18.651 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:18.651 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:18.651 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.650+0000 7fa1ecd68640 4 set_mon_vals no callback set 2024-03-25T19:46:18.652 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.651+0000 7fa1deffd640 1 -- 172.21.15.81:0/2351629771 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fa1d8004130 con 0x7fa1e813f710 2024-03-25T19:46:18.652 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.652+0000 7fa1ee56b640 10 monclient: get_auth_request con 0x7fa1cc06dd40 auth_method 0 2024-03-25T19:46:18.652 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.652+0000 7fa1deffd640 10 client.4425.objecter ms_handle_connect 0x7fa1cc06dd40 2024-03-25T19:46:18.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.657+0000 7fa1deffd640 1 -- 172.21.15.81:0/2351629771 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fa1d8054a20 con 0x7fa1e813f710 2024-03-25T19:46:18.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.657+0000 7fa1deffd640 10 client.4425.objecter ms_dispatch 0x7fa1e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:18.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.657+0000 7fa1deffd640 3 client.4425.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:18.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.657+0000 7fa1deffd640 3 client.4425.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:18.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.657+0000 7fa1deffd640 20 client.4425.objecter dump_active .. 0 homeless 2024-03-25T19:46:18.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.657+0000 7fa1deffd640 1 -- 172.21.15.81:0/2351629771 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa1d8050530 con 0x7fa1e813f710 2024-03-25T19:46:18.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.657+0000 7fa1deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:18.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.657+0000 7fa1deffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:18.869 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.869+0000 7fa1f07cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:18.869 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.869+0000 7fa1f07cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:18.869 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.869+0000 7fa1f07cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:18.869 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.869+0000 7fa1f07cf640 1 -- 172.21.15.81:0/2351629771 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1e8069f40 con 0x7fa1e813f710 2024-03-25T19:46:18.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.893+0000 7fa1deffd640 1 -- 172.21.15.81:0/2351629771 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fa1d8004460 con 0x7fa1e813f710 2024-03-25T19:46:18.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.893+0000 7fa1deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:18.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.893+0000 7fa1deffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:18.893 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:46:18.894 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:18.895 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.895+0000 7fa1f07cf640 10 librados: watch_flush enter 2024-03-25T19:46:18.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.895+0000 7fa1f07cf640 10 librados: watch_flush exit 2024-03-25T19:46:18.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.895+0000 7fa1f07cf640 20 client.4425.objecter shutdown clearing up homeless session... 2024-03-25T19:46:18.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.895+0000 7fa1f07cf640 10 client.4425.objecter successfully canceled tick 2024-03-25T19:46:18.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.896+0000 7fa1f07cf640 1 -- 172.21.15.81:0/2351629771 >> v1:172.21.15.98:6800/3980829775 conn(0x7fa1cc06dd40 legacy=0x7fa1cc070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:18.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.896+0000 7fa1f07cf640 10 monclient: shutdown 2024-03-25T19:46:18.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.896+0000 7fa1f07cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:18.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.896+0000 7fa1f07cf640 1 -- 172.21.15.81:0/2351629771 >> v1:172.21.15.81:6789/0 conn(0x7fa1e813f710 legacy=0x7fa1e81daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:18.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.896+0000 7fa1f07cf640 1 -- 172.21.15.81:0/2351629771 shutdown_connections 2024-03-25T19:46:18.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.896+0000 7fa1f07cf640 1 -- 172.21.15.81:0/2351629771 >> 172.21.15.81:0/2351629771 conn(0x7fa1e80b7ea0 msgr2=0x7fa1e8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:18.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.896+0000 7fa1f07cf640 1 -- 172.21.15.81:0/2351629771 shutdown_connections 2024-03-25T19:46:18.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.896+0000 7fa1f07cf640 1 -- 172.21.15.81:0/2351629771 wait complete. 2024-03-25T19:46:18.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:18.896+0000 7fa1f07cf640 1 librados: shutdown 2024-03-25T19:46:18.908 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:25.715 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:25.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.830+0000 7f822a3c6640 10 monclient: get_monmap_and_config 2024-03-25T19:46:25.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.830+0000 7f822a3c6640 10 monclient: build_initial_monmap 2024-03-25T19:46:25.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.830+0000 7f822a3c6640 10 monclient: monmap: 2024-03-25T19:46:25.831 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:46:25.831 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:25.831 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:46:25.831795+0000 2024-03-25T19:46:25.831 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:46:25.831795+0000 2024-03-25T19:46:25.831 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:25.831 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:25.831 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:46:25.831 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:46:25.831 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:46:25.831 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:25.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.831+0000 7f822a3c6640 1 Processor -- start 2024-03-25T19:46:25.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.831+0000 7f822a3c6640 1 -- start start 2024-03-25T19:46:25.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.831+0000 7f822a3c6640 10 monclient: init 2024-03-25T19:46:25.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.832+0000 7f822a3c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:25.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.832+0000 7f822a3c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:25.833 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.832+0000 7f822a3c6640 10 monclient(hunting): picked mon.noname-a con 0x7f82240a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:46:25.833 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.832+0000 7f822a3c6640 10 monclient(hunting): picked mon.noname-c con 0x7f822413f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:46:25.833 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.832+0000 7f822a3c6640 10 monclient(hunting): picked mon.noname-b con 0x7f82241430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:46:25.833 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.832+0000 7f822a3c6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8224142520 con 0x7f82240a98f0 2024-03-25T19:46:25.833 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.832+0000 7f822a3c6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82240b28a0 con 0x7f822413f710 2024-03-25T19:46:25.833 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.832+0000 7f822a3c6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82240b2a30 con 0x7f82241430c0 2024-03-25T19:46:25.833 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.832+0000 7f822a3c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:25.833 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.832+0000 7f822a3c6640 10 monclient(hunting): authenticate will time out at 4220.914448s 2024-03-25T19:46:25.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.848+0000 7f8223fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f82240a98f0 0x7f82240a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:50952/0 (socket says 172.21.15.81:50952) 2024-03-25T19:46:25.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.848+0000 7f8223fff640 1 -- 172.21.15.81:0/30845021 learned_addr learned my addr 172.21.15.81:0/30845021 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:46:25.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.859+0000 7f8222ffd640 1 -- 172.21.15.81:0/30845021 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381345271 0 0) 0x7f8224142520 con 0x7f82240a98f0 2024-03-25T19:46:25.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.859+0000 7f8222ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:25.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.859+0000 7f8222ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:25.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.859+0000 7f8222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:25.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.859+0000 7f8222ffd640 10 monclient(hunting): my global_id is 4428 2024-03-25T19:46:25.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.859+0000 7f8222ffd640 1 -- 172.21.15.81:0/30845021 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f820c003520 con 0x7f82240a98f0 2024-03-25T19:46:25.868 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.869+0000 7f8222ffd640 1 -- 172.21.15.81:0/30845021 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864869673 0 0) 0x7f82240b28a0 con 0x7f822413f710 2024-03-25T19:46:25.869 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.869+0000 7f8222ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:25.869 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.869+0000 7f8222ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:25.869 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.869+0000 7f8222ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:25.869 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.869+0000 7f8222ffd640 10 monclient(hunting): my global_id is 4426 2024-03-25T19:46:25.869 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.869+0000 7f8222ffd640 1 -- 172.21.15.81:0/30845021 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8224142520 con 0x7f822413f710 2024-03-25T19:46:25.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f8222ffd640 1 -- 172.21.15.81:0/30845021 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1701748486 0 0) 0x7f8224142520 con 0x7f822413f710 2024-03-25T19:46:25.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f8222ffd640 1 -- 172.21.15.81:0/30845021 >> v1:172.21.15.81:6790/0 conn(0x7f82241430c0 legacy=0x7f822414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:25.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f8222ffd640 1 -- 172.21.15.81:0/30845021 >> v1:172.21.15.81:6789/0 conn(0x7f82240a98f0 legacy=0x7f82240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:25.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f8222ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:25.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f8222ffd640 1 monclient: found mon.noname-c 2024-03-25T19:46:25.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f8222ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:25.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f8222ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:46:25.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f8222ffd640 1 -- 172.21.15.81:0/30845021 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82240b44c0 con 0x7f822413f710 2024-03-25T19:46:25.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f8222ffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:25.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f8222ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:25.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f8222ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:25.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f8222ffd640 1 -- 172.21.15.81:0/30845021 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8218001670 con 0x7f822413f710 2024-03-25T19:46:25.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f822a3c6640 5 monclient: authenticate success, global_id 4426 2024-03-25T19:46:25.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f822a3c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:25.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f8222ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:25.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f8222ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:25.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f8222ffd640 10 monclient: dump: 2024-03-25T19:46:25.876 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:25.876 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:25.877 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:25.877 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:25.877 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:25.877 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:25.877 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:25.877 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:25.877 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:25.877 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:25.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.875+0000 7f822a3c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:25.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.880+0000 7f8222ffd640 1 -- 172.21.15.81:0/30845021 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82180017d0 con 0x7f822413f710 2024-03-25T19:46:25.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.880+0000 7f8222ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:25.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.880+0000 7f8222ffd640 1 -- 172.21.15.81:0/30845021 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8218001ba0 con 0x7f822413f710 2024-03-25T19:46:25.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.880+0000 7f8222ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:25.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.880+0000 7f8222ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:25.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.880+0000 7f8222ffd640 10 monclient: dump: 2024-03-25T19:46:25.880 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:25.880 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:25.880 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:25.880 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:25.880 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:25.881 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:25.881 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:25.881 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:25.881 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:25.881 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:25.881 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.880+0000 7f822a3c6640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:25.881 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.880+0000 7f822a3c6640 4 set_mon_vals no callback set 2024-03-25T19:46:25.882 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.881+0000 7f822a3c6640 10 monclient: shutdown 2024-03-25T19:46:25.882 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.881+0000 7f822a3c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:25.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.881+0000 7f822a3c6640 1 -- 172.21.15.81:0/30845021 >> v1:172.21.15.98:6789/0 conn(0x7f822413f710 legacy=0x7f82241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:25.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.881+0000 7f822a3c6640 1 -- 172.21.15.81:0/30845021 shutdown_connections 2024-03-25T19:46:25.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.881+0000 7f822a3c6640 1 -- 172.21.15.81:0/30845021 >> 172.21.15.81:0/30845021 conn(0x7f82240b7ea0 msgr2=0x7f82240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:25.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.881+0000 7f822a3c6640 1 -- 172.21.15.81:0/30845021 shutdown_connections 2024-03-25T19:46:25.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.881+0000 7f822a3c6640 1 -- 172.21.15.81:0/30845021 wait complete. 2024-03-25T19:46:25.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.882+0000 7f822a3c6640 10 monclient: build_initial_monmap 2024-03-25T19:46:25.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.882+0000 7f822a3c6640 10 monclient: monmap: 2024-03-25T19:46:25.883 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:46:25.883 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:25.883 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:46:25.883 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:46:25.883 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:25.883 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:46:25.883 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:46:25.883 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:46:25.884 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:46:25.884 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:25.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.882+0000 7f822a3c6640 1 librados: starting msgr at 2024-03-25T19:46:25.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.882+0000 7f822a3c6640 1 librados: starting objecter 2024-03-25T19:46:25.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.882+0000 7f822a3c6640 1 Processor -- start 2024-03-25T19:46:25.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.883+0000 7f822a3c6640 1 -- start start 2024-03-25T19:46:25.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.883+0000 7f822a3c6640 1 librados: setting wanted keys 2024-03-25T19:46:25.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.883+0000 7f822a3c6640 1 librados: calling monclient init 2024-03-25T19:46:25.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.883+0000 7f822a3c6640 10 monclient: init 2024-03-25T19:46:25.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.883+0000 7f822a3c6640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:25.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.883+0000 7f822a3c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:25.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.883+0000 7f822a3c6640 10 monclient(hunting): picked mon.noname-b con 0x7f82240a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:46:25.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.883+0000 7f822a3c6640 10 monclient(hunting): picked mon.noname-a con 0x7f822413f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:46:25.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.884+0000 7f822a3c6640 10 monclient(hunting): picked mon.noname-c con 0x7f82241430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:46:25.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.884+0000 7f822a3c6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82240abae0 con 0x7f822413f710 2024-03-25T19:46:25.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.884+0000 7f822a3c6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82240acb60 con 0x7f82240a98f0 2024-03-25T19:46:25.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.884+0000 7f822a3c6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82240abd50 con 0x7f82241430c0 2024-03-25T19:46:25.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.884+0000 7f822a3c6640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:25.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.884+0000 7f822a3c6640 10 monclient(hunting): authenticate will time out at 4220.965946s 2024-03-25T19:46:25.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.893+0000 7f8228963640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f82241430c0 0x7f82241e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:53308/0 (socket says 172.21.15.81:53308) 2024-03-25T19:46:25.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.893+0000 7f8228963640 1 -- 172.21.15.81:0/3167032318 learned_addr learned my addr 172.21.15.81:0/3167032318 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:46:25.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.895+0000 7f8220ff9640 10 client.?.objecter ms_handle_connect 0x7f82240a98f0 2024-03-25T19:46:25.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.895+0000 7f8220ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:25.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.895+0000 7f8220ff9640 10 client.?.objecter ms_handle_connect 0x7f82241430c0 2024-03-25T19:46:25.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.895+0000 7f8220ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:25.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.905+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707056085 0 0) 0x7f82240acb60 con 0x7f82240a98f0 2024-03-25T19:46:25.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.905+0000 7f8220ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:25.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.905+0000 7f8220ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:25.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.905+0000 7f8220ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:25.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.905+0000 7f8220ff9640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:46:25.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.905+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8210004eb0 con 0x7f82240a98f0 2024-03-25T19:46:25.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.907+0000 7f8220ff9640 10 client.?.objecter ms_handle_connect 0x7f822413f710 2024-03-25T19:46:25.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.907+0000 7f8220ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:25.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.908+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679052001 0 0) 0x7f82240abd50 con 0x7f82241430c0 2024-03-25T19:46:25.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.908+0000 7f8220ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:25.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.908+0000 7f8220ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:25.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.908+0000 7f8220ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:25.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.908+0000 7f8220ff9640 10 monclient(hunting): my global_id is 4415 2024-03-25T19:46:25.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.908+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82240acb60 con 0x7f82241430c0 2024-03-25T19:46:25.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.933+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082667694 0 0) 0x7f82240abae0 con 0x7f822413f710 2024-03-25T19:46:25.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.933+0000 7f8220ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:25.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.933+0000 7f8220ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:25.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.933+0000 7f8220ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:25.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.933+0000 7f8220ff9640 10 monclient(hunting): my global_id is 4431 2024-03-25T19:46:25.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.933+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82240abd50 con 0x7f822413f710 2024-03-25T19:46:25.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.933+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3048099948 0 0) 0x7f8210004eb0 con 0x7f82240a98f0 2024-03-25T19:46:25.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.933+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82240abae0 con 0x7f82240a98f0 2024-03-25T19:46:25.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.933+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8214001950 con 0x7f82240a98f0 2024-03-25T19:46:25.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.933+0000 7f8220ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:25.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.933+0000 7f8220ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:25.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.933+0000 7f8220ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:25.934 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:25.934 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:25.934 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:25.934 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:25.934 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:25.934 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:25.934 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:25.934 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:25.934 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:25.934 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:25.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.933+0000 7f8220ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:25.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.936+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2090995930 0 0) 0x7f82240acb60 con 0x7f82241430c0 2024-03-25T19:46:25.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.936+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8210004eb0 con 0x7f82241430c0 2024-03-25T19:46:25.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.936+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f821c001c10 con 0x7f82241430c0 2024-03-25T19:46:25.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.936+0000 7f8220ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:25.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.936+0000 7f8220ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:25.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.936+0000 7f8220ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:25.936 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:25.936 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:25.936 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:25.936 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:25.936 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:25.936 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:25.936 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:25.936 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:25.936 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:25.936 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:25.956 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4290253848 0 0) 0x7f82240abae0 con 0x7f82240a98f0 2024-03-25T19:46:25.956 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 >> v1:172.21.15.81:6790/0 conn(0x7f82241430c0 legacy=0x7f82241e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:25.956 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 >> v1:172.21.15.81:6789/0 conn(0x7f822413f710 legacy=0x7f82240ac450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:25.956 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f8220ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:46:25.956 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f8220ff9640 1 monclient: found mon.b 2024-03-25T19:46:25.957 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f8220ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:25.957 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f8220ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:46:25.957 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82240ac030 con 0x7f82240a98f0 2024-03-25T19:46:25.957 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f8220ff9640 10 monclient: _finish_auth 0 2024-03-25T19:46:25.957 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f8220ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:46:25.957 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f8220ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:25.957 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f822a3c6640 5 monclient: authenticate success, global_id 4429 2024-03-25T19:46:25.957 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f822a3c6640 10 monclient: _renew_subs 2024-03-25T19:46:25.957 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f822a3c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:46:25.957 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f822a3c6640 1 -- 172.21.15.81:0/3167032318 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82241390d0 con 0x7f82240a98f0 2024-03-25T19:46:25.957 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f822a3c6640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:25.957 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f822a3c6640 10 monclient: _renew_subs 2024-03-25T19:46:25.957 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f822a3c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:46:25.958 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f822a3c6640 1 -- 172.21.15.81:0/3167032318 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8224136520 con 0x7f82240a98f0 2024-03-25T19:46:25.958 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.956+0000 7f822a3c6640 1 librados: init done 2024-03-25T19:46:25.958 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.957+0000 7f822a3c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:25.958 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.957+0000 7f822a3c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:25.958 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.957+0000 7f822a3c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:46:25.958 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:25.957+0000 7f822a3c6640 1 -- 172.21.15.81:0/3167032318 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82240627b0 con 0x7f82240a98f0 2024-03-25T19:46:26.244 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.244+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8214001d60 con 0x7f82240a98f0 2024-03-25T19:46:26.244 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.244+0000 7f8220ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:26.244 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.244+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f8214004980 con 0x7f82240a98f0 2024-03-25T19:46:26.245 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.244+0000 7f8220ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:26.245 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.244+0000 7f8220ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:26.245 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.244+0000 7f8220ff9640 10 monclient: dump: 2024-03-25T19:46:26.245 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:26.245 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:26.245 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:26.245 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:26.245 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:26.245 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:26.245 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:26.245 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:26.245 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:26.245 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:26.245 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.244+0000 7f82227fc640 4 set_mon_vals no callback set 2024-03-25T19:46:26.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.245+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f821401a9d0 con 0x7f82240a98f0 2024-03-25T19:46:26.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.246+0000 7f82237fe640 10 monclient: get_auth_request con 0x7f821006df20 auth_method 0 2024-03-25T19:46:26.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.246+0000 7f8220ff9640 10 client.4429.objecter ms_handle_connect 0x7f821006df20 2024-03-25T19:46:26.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.252+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f82140547b0 con 0x7f82240a98f0 2024-03-25T19:46:26.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.252+0000 7f8220ff9640 10 client.4429.objecter ms_dispatch 0x7f8224000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:26.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.252+0000 7f8220ff9640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:26.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.252+0000 7f8220ff9640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:26.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.252+0000 7f8220ff9640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-25T19:46:26.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.253+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82140500f0 con 0x7f82240a98f0 2024-03-25T19:46:26.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.253+0000 7f8220ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:26.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.253+0000 7f8220ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:26.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.505+0000 7f822a3c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:26.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.505+0000 7f822a3c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:26.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.505+0000 7f822a3c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:46:26.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.505+0000 7f822a3c6640 1 -- 172.21.15.81:0/3167032318 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f822413ce70 con 0x7f82240a98f0 2024-03-25T19:46:26.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.641+0000 7f8220ff9640 1 -- 172.21.15.81:0/3167032318 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f82140521f0 con 0x7f82240a98f0 2024-03-25T19:46:26.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.641+0000 7f8220ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:26.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.641+0000 7f8220ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:26.642 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:46:26.642 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:26.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.643+0000 7f822a3c6640 10 librados: watch_flush enter 2024-03-25T19:46:26.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.643+0000 7f822a3c6640 10 librados: watch_flush exit 2024-03-25T19:46:26.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.643+0000 7f822a3c6640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-25T19:46:26.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.643+0000 7f822a3c6640 10 client.4429.objecter successfully canceled tick 2024-03-25T19:46:26.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.644+0000 7f822a3c6640 1 -- 172.21.15.81:0/3167032318 >> v1:172.21.15.98:6800/3980829775 conn(0x7f821006df20 legacy=0x7f82100703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:26.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.644+0000 7f822a3c6640 10 monclient: shutdown 2024-03-25T19:46:26.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.644+0000 7f822a3c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:26.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.644+0000 7f822a3c6640 1 -- 172.21.15.81:0/3167032318 >> v1:172.21.15.98:6789/0 conn(0x7f82240a98f0 legacy=0x7f8224138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:26.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.644+0000 7f822a3c6640 1 -- 172.21.15.81:0/3167032318 shutdown_connections 2024-03-25T19:46:26.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.644+0000 7f822a3c6640 1 -- 172.21.15.81:0/3167032318 >> 172.21.15.81:0/3167032318 conn(0x7f82240b7ea0 msgr2=0x7f8224074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:26.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.644+0000 7f822a3c6640 1 -- 172.21.15.81:0/3167032318 shutdown_connections 2024-03-25T19:46:26.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.644+0000 7f822a3c6640 1 -- 172.21.15.81:0/3167032318 wait complete. 2024-03-25T19:46:26.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:26.644+0000 7f822a3c6640 1 librados: shutdown 2024-03-25T19:46:26.653 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:33.460 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:33.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.566+0000 7f467897b640 10 monclient: get_monmap_and_config 2024-03-25T19:46:33.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.566+0000 7f467897b640 10 monclient: build_initial_monmap 2024-03-25T19:46:33.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.566+0000 7f467897b640 10 monclient: monmap: 2024-03-25T19:46:33.566 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:46:33.566 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:33.566 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:46:33.567137+0000 2024-03-25T19:46:33.566 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:46:33.567137+0000 2024-03-25T19:46:33.566 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:33.566 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:33.566 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:46:33.566 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:46:33.566 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:46:33.566 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:33.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.566+0000 7f467897b640 1 Processor -- start 2024-03-25T19:46:33.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.567+0000 7f467897b640 1 -- start start 2024-03-25T19:46:33.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.567+0000 7f467897b640 10 monclient: init 2024-03-25T19:46:33.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.567+0000 7f467897b640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:33.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.567+0000 7f467897b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:33.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.567+0000 7f467897b640 10 monclient(hunting): picked mon.noname-b con 0x7f46740b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:46:33.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.567+0000 7f467897b640 10 monclient(hunting): picked mon.noname-c con 0x7f46740b1730 addr v1:172.21.15.98:6789/0 2024-03-25T19:46:33.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.567+0000 7f467897b640 10 monclient(hunting): picked mon.noname-a con 0x7f467414be90 addr v1:172.21.15.81:6789/0 2024-03-25T19:46:33.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.567+0000 7f467897b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46740af940 con 0x7f467414be90 2024-03-25T19:46:33.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.567+0000 7f467897b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46740b0a70 con 0x7f46740b1730 2024-03-25T19:46:33.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.567+0000 7f467897b640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46740b0d50 con 0x7f46740b4d60 2024-03-25T19:46:33.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.567+0000 7f467897b640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:33.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.567+0000 7f467897b640 10 monclient(hunting): authenticate will time out at 4228.649426s 2024-03-25T19:46:33.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.568+0000 7f467259c640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f46740b4d60 0x7f467413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:55954/0 (socket says 172.21.15.81:55954) 2024-03-25T19:46:33.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.568+0000 7f467259c640 1 -- 172.21.15.81:0/1363025704 learned_addr learned my addr 172.21.15.81:0/1363025704 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:46:33.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.621+0000 7f4671d9b640 1 -- 172.21.15.81:0/1363025704 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170800130 0 0) 0x7f46740af940 con 0x7f467414be90 2024-03-25T19:46:33.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.621+0000 7f4671d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:46:33.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.621+0000 7f4671d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:33.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.621+0000 7f4671d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:33.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.621+0000 7f4671d9b640 10 monclient(hunting): my global_id is 4434 2024-03-25T19:46:33.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.621+0000 7f4671d9b640 1 -- 172.21.15.81:0/1363025704 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4660003520 con 0x7f467414be90 2024-03-25T19:46:33.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.640+0000 7f4671d9b640 1 -- 172.21.15.81:0/1363025704 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1429852506 0 0) 0x7f4660003520 con 0x7f467414be90 2024-03-25T19:46:33.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.640+0000 7f4671d9b640 1 -- 172.21.15.81:0/1363025704 >> v1:172.21.15.81:6790/0 conn(0x7f46740b4d60 legacy=0x7f467413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:33.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.640+0000 7f4671d9b640 1 -- 172.21.15.81:0/1363025704 >> v1:172.21.15.98:6789/0 conn(0x7f46740b1730 legacy=0x7f46740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:33.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.640+0000 7f4671d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:46:33.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.640+0000 7f4671d9b640 1 monclient: found mon.noname-a 2024-03-25T19:46:33.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.640+0000 7f4671d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:33.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.640+0000 7f4671d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:46:33.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.640+0000 7f4671d9b640 1 -- 172.21.15.81:0/1363025704 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46740b1030 con 0x7f467414be90 2024-03-25T19:46:33.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.640+0000 7f4671d9b640 10 monclient: _finish_auth 0 2024-03-25T19:46:33.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.640+0000 7f4671d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:46:33.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.640+0000 7f4671d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:33.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.640+0000 7f4671d9b640 1 -- 172.21.15.81:0/1363025704 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f466c001a10 con 0x7f467414be90 2024-03-25T19:46:33.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.640+0000 7f4671d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:33.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.640+0000 7f4671d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:33.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.640+0000 7f4671d9b640 10 monclient: dump: 2024-03-25T19:46:33.641 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:33.642 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:33.642 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:33.642 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:33.642 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:33.642 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:33.642 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:33.642 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:33.642 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:33.642 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:33.642 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.640+0000 7f467897b640 5 monclient: authenticate success, global_id 4434 2024-03-25T19:46:33.642 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.640+0000 7f467897b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:33.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.656+0000 7f4671d9b640 1 -- 172.21.15.81:0/1363025704 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f466c001b70 con 0x7f467414be90 2024-03-25T19:46:33.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.656+0000 7f4671d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:33.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.656+0000 7f4671d9b640 1 -- 172.21.15.81:0/1363025704 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f466c001f80 con 0x7f467414be90 2024-03-25T19:46:33.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.656+0000 7f4671d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:33.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.656+0000 7f4671d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:33.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.656+0000 7f4671d9b640 10 monclient: dump: 2024-03-25T19:46:33.656 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:33.656 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:33.657 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:33.657 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:33.657 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:33.657 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:33.657 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:33.657 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:33.657 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:33.657 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:33.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.656+0000 7f467897b640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:33.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.656+0000 7f467897b640 4 set_mon_vals no callback set 2024-03-25T19:46:33.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.657+0000 7f467897b640 10 monclient: shutdown 2024-03-25T19:46:33.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.657+0000 7f467897b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:33.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.657+0000 7f467897b640 1 -- 172.21.15.81:0/1363025704 >> v1:172.21.15.81:6789/0 conn(0x7f467414be90 legacy=0x7f467414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:33.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.657+0000 7f467897b640 1 -- 172.21.15.81:0/1363025704 shutdown_connections 2024-03-25T19:46:33.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.657+0000 7f467897b640 1 -- 172.21.15.81:0/1363025704 >> 172.21.15.81:0/1363025704 conn(0x7f46740617a0 msgr2=0x7f4674061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:33.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.657+0000 7f467897b640 1 -- 172.21.15.81:0/1363025704 shutdown_connections 2024-03-25T19:46:33.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.657+0000 7f467897b640 1 -- 172.21.15.81:0/1363025704 wait complete. 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.658+0000 7f467897b640 10 monclient: build_initial_monmap 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.658+0000 7f467897b640 10 monclient: monmap: 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.659+0000 7f467897b640 1 librados: starting msgr at 2024-03-25T19:46:33.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.659+0000 7f467897b640 1 librados: starting objecter 2024-03-25T19:46:33.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.659+0000 7f467897b640 1 Processor -- start 2024-03-25T19:46:33.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.659+0000 7f467897b640 1 -- start start 2024-03-25T19:46:33.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.659+0000 7f467897b640 1 librados: setting wanted keys 2024-03-25T19:46:33.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.659+0000 7f467897b640 1 librados: calling monclient init 2024-03-25T19:46:33.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.659+0000 7f467897b640 10 monclient: init 2024-03-25T19:46:33.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.659+0000 7f467897b640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:33.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.659+0000 7f467897b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:33.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.659+0000 7f467897b640 10 monclient(hunting): picked mon.noname-a con 0x7f46740b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:46:33.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.659+0000 7f467897b640 10 monclient(hunting): picked mon.noname-c con 0x7f46740b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:46:33.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.659+0000 7f467897b640 10 monclient(hunting): picked mon.noname-b con 0x7f467414be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:46:33.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.659+0000 7f467897b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4674140380 con 0x7f46740b1730 2024-03-25T19:46:33.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.659+0000 7f467897b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4674140660 con 0x7f467414be90 2024-03-25T19:46:33.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.659+0000 7f467897b640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4674141e30 con 0x7f46740b4d60 2024-03-25T19:46:33.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.659+0000 7f467897b640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:33.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.659+0000 7f467897b640 10 monclient(hunting): authenticate will time out at 4228.741805s 2024-03-25T19:46:33.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.660+0000 7f4672d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f467414be90 0x7f46741e2740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:56208/0 (socket says 172.21.15.81:56208) 2024-03-25T19:46:33.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.660+0000 7f4672d9d640 1 -- 172.21.15.81:0/2070638156 learned_addr learned my addr 172.21.15.81:0/2070638156 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:46:33.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.661+0000 7f466affd640 10 client.?.objecter ms_handle_connect 0x7f467414be90 2024-03-25T19:46:33.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.661+0000 7f466affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:33.671 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.672+0000 7f466affd640 10 client.?.objecter ms_handle_connect 0x7f46740b1730 2024-03-25T19:46:33.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.672+0000 7f466affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:33.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.676+0000 7f466affd640 10 client.?.objecter ms_handle_connect 0x7f46740b4d60 2024-03-25T19:46:33.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.676+0000 7f466affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:33.687 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.688+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295074062 0 0) 0x7f4674140380 con 0x7f46740b1730 2024-03-25T19:46:33.687 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.688+0000 7f466affd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:33.688 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.688+0000 7f466affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:33.688 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.688+0000 7f466affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:33.688 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.688+0000 7f466affd640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:46:33.688 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.688+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4654004eb0 con 0x7f46740b1730 2024-03-25T19:46:33.744 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.744+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418767424 0 0) 0x7f4674140660 con 0x7f467414be90 2024-03-25T19:46:33.744 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.744+0000 7f466affd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:33.744 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.744+0000 7f466affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:33.744 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.744+0000 7f466affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:33.744 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.744+0000 7f466affd640 10 monclient(hunting): my global_id is 4435 2024-03-25T19:46:33.744 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.744+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4674140380 con 0x7f467414be90 2024-03-25T19:46:33.756 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.757+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2911511324 0 0) 0x7f4654004eb0 con 0x7f46740b1730 2024-03-25T19:46:33.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.757+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4674140660 con 0x7f46740b1730 2024-03-25T19:46:33.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.757+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f46640018a0 con 0x7f46740b1730 2024-03-25T19:46:33.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.757+0000 7f466affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:33.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.757+0000 7f466affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:33.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.757+0000 7f466affd640 10 monclient(hunting): dump: 2024-03-25T19:46:33.757 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:33.757 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:33.757 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:33.757 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:33.757 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:33.757 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:33.757 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:33.758 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:33.758 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:33.758 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:33.758 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.757+0000 7f466affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:33.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.760+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593301512 0 0) 0x7f4674141e30 con 0x7f46740b4d60 2024-03-25T19:46:33.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.760+0000 7f466affd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:33.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.760+0000 7f466affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:33.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.760+0000 7f466affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:33.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.760+0000 7f466affd640 10 monclient(hunting): my global_id is 4421 2024-03-25T19:46:33.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.760+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4654004eb0 con 0x7f46740b4d60 2024-03-25T19:46:33.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 66918329 0 0) 0x7f4674140660 con 0x7f46740b1730 2024-03-25T19:46:33.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 >> v1:172.21.15.81:6790/0 conn(0x7f46740b4d60 legacy=0x7f4674142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:33.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 >> v1:172.21.15.98:6789/0 conn(0x7f467414be90 legacy=0x7f46741e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:33.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f466affd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:33.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f466affd640 1 monclient: found mon.a 2024-03-25T19:46:33.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f466affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:33.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f466affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:33.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4674142110 con 0x7f46740b1730 2024-03-25T19:46:33.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f466affd640 10 monclient: _finish_auth 0 2024-03-25T19:46:33.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f466affd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:33.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f466affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:33.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f467897b640 5 monclient: authenticate success, global_id 4437 2024-03-25T19:46:33.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f467897b640 10 monclient: _renew_subs 2024-03-25T19:46:33.773 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f467897b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:33.773 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f467897b640 1 -- 172.21.15.81:0/2070638156 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4674140e60 con 0x7f46740b1730 2024-03-25T19:46:33.773 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f467897b640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:33.773 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f467897b640 10 monclient: _renew_subs 2024-03-25T19:46:33.773 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f467897b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:33.773 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f467897b640 1 -- 172.21.15.81:0/2070638156 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4674142c40 con 0x7f46740b1730 2024-03-25T19:46:33.773 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.772+0000 7f467897b640 1 librados: init done 2024-03-25T19:46:33.774 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.773+0000 7f467897b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:33.774 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.773+0000 7f467897b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:33.774 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.773+0000 7f467897b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:33.774 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:33.773+0000 7f467897b640 1 -- 172.21.15.81:0/2070638156 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46740627b0 con 0x7f46740b1730 2024-03-25T19:46:34.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.076+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4664001d10 con 0x7f46740b1730 2024-03-25T19:46:34.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.076+0000 7f466affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:34.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.076+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4664004980 con 0x7f46740b1730 2024-03-25T19:46:34.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.076+0000 7f466affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:34.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.076+0000 7f466affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:34.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.076+0000 7f466affd640 10 monclient: dump: 2024-03-25T19:46:34.076 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:34.076 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:34.076 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:34.077 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:34.077 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:34.077 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:34.077 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:34.077 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:34.077 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:34.077 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:34.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.076+0000 7f467159a640 4 set_mon_vals no callback set 2024-03-25T19:46:34.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.077+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f4664004ba0 con 0x7f46740b1730 2024-03-25T19:46:34.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.077+0000 7f466bfff640 10 monclient: get_auth_request con 0x7f465406de00 auth_method 0 2024-03-25T19:46:34.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.078+0000 7f466affd640 10 client.4437.objecter ms_handle_connect 0x7f465406de00 2024-03-25T19:46:34.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.084+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f4664054760 con 0x7f46740b1730 2024-03-25T19:46:34.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.084+0000 7f466affd640 10 client.4437.objecter ms_dispatch 0x7f4674000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:34.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.084+0000 7f466affd640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:34.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.084+0000 7f466affd640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:34.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.084+0000 7f466affd640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-25T19:46:34.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.084+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46640500a0 con 0x7f46740b1730 2024-03-25T19:46:34.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.084+0000 7f466affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:34.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.084+0000 7f466affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:34.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.337+0000 7f467897b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:34.337 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.337+0000 7f467897b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:34.337 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.337+0000 7f467897b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:34.337 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.337+0000 7f467897b640 1 -- 172.21.15.81:0/2070638156 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46740b5cf0 con 0x7f46740b1730 2024-03-25T19:46:34.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.381+0000 7f466affd640 1 -- 172.21.15.81:0/2070638156 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f46640521a0 con 0x7f46740b1730 2024-03-25T19:46:34.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.381+0000 7f466affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:34.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.381+0000 7f466affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:34.381 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:46:34.381 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:34.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.383+0000 7f467897b640 10 librados: watch_flush enter 2024-03-25T19:46:34.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.383+0000 7f467897b640 10 librados: watch_flush exit 2024-03-25T19:46:34.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.383+0000 7f467897b640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-25T19:46:34.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.383+0000 7f467897b640 10 client.4437.objecter successfully canceled tick 2024-03-25T19:46:34.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.383+0000 7f467897b640 1 -- 172.21.15.81:0/2070638156 >> v1:172.21.15.98:6800/3980829775 conn(0x7f465406de00 legacy=0x7f46540702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:34.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.383+0000 7f467897b640 10 monclient: shutdown 2024-03-25T19:46:34.384 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.383+0000 7f467897b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:34.384 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.383+0000 7f467897b640 1 -- 172.21.15.81:0/2070638156 >> v1:172.21.15.81:6789/0 conn(0x7f46740b1730 legacy=0x7f4674138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:34.384 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.383+0000 7f467897b640 1 -- 172.21.15.81:0/2070638156 shutdown_connections 2024-03-25T19:46:34.384 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.383+0000 7f467897b640 1 -- 172.21.15.81:0/2070638156 >> 172.21.15.81:0/2070638156 conn(0x7f46740617a0 msgr2=0x7f46740b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:34.384 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.383+0000 7f467897b640 1 -- 172.21.15.81:0/2070638156 shutdown_connections 2024-03-25T19:46:34.384 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.383+0000 7f467897b640 1 -- 172.21.15.81:0/2070638156 wait complete. 2024-03-25T19:46:34.384 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:34.384+0000 7f467897b640 1 librados: shutdown 2024-03-25T19:46:34.395 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:37.371 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:05:00.012 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:37.378 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:05:00.020 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:37.405 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:05:00.047 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:37.421 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:05:00.063 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:37.429 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:05:00.071 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:37.432 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:00.067 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:37.443 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:00.077 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:37.451 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:05:00.093 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:37.456 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:00.090 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:41.202 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:41.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.317+0000 7f292db5d640 10 monclient: get_monmap_and_config 2024-03-25T19:46:41.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.317+0000 7f292db5d640 10 monclient: build_initial_monmap 2024-03-25T19:46:41.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.317+0000 7f292db5d640 10 monclient: monmap: 2024-03-25T19:46:41.317 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:46:41.317 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:41.317 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:46:41.318355+0000 2024-03-25T19:46:41.317 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:46:41.318355+0000 2024-03-25T19:46:41.317 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:41.317 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:41.318 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:46:41.318 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:46:41.318 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:46:41.318 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:41.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.318+0000 7f292db5d640 1 Processor -- start 2024-03-25T19:46:41.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.318+0000 7f292db5d640 1 -- start start 2024-03-25T19:46:41.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.318+0000 7f292db5d640 10 monclient: init 2024-03-25T19:46:41.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.319+0000 7f292db5d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:41.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.319+0000 7f292db5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:41.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.319+0000 7f292db5d640 10 monclient(hunting): picked mon.noname-a con 0x7f29280b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:46:41.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.319+0000 7f292db5d640 10 monclient(hunting): picked mon.noname-c con 0x7f29280b1730 addr v1:172.21.15.98:6789/0 2024-03-25T19:46:41.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.319+0000 7f292db5d640 10 monclient(hunting): picked mon.noname-b con 0x7f292814be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:46:41.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.319+0000 7f292db5d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29280af940 con 0x7f29280b4d60 2024-03-25T19:46:41.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.319+0000 7f292db5d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29280b0a70 con 0x7f29280b1730 2024-03-25T19:46:41.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.319+0000 7f292db5d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29280b0d50 con 0x7f292814be90 2024-03-25T19:46:41.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.319+0000 7f292db5d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:41.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.319+0000 7f292db5d640 10 monclient(hunting): authenticate will time out at 4236.401081s 2024-03-25T19:46:41.331 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.331+0000 7f29277fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f29280b4d60 0x7f292813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:55732/0 (socket says 172.21.15.81:55732) 2024-03-25T19:46:41.331 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.331+0000 7f2926ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f29280b1730 0x7f29280b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:33090/0 (socket says 172.21.15.81:33090) 2024-03-25T19:46:41.331 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.331+0000 7f29277fe640 1 -- 172.21.15.81:0/3082514548 learned_addr learned my addr 172.21.15.81:0/3082514548 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:46:41.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.350+0000 7f29267fc640 1 -- 172.21.15.81:0/3082514548 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299076463 0 0) 0x7f29280af940 con 0x7f29280b4d60 2024-03-25T19:46:41.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.350+0000 7f29267fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:41.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.350+0000 7f29267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:41.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.350+0000 7f29267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:41.349 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.350+0000 7f29267fc640 10 monclient(hunting): my global_id is 4440 2024-03-25T19:46:41.350 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.350+0000 7f29267fc640 1 -- 172.21.15.81:0/3082514548 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2910003520 con 0x7f29280b4d60 2024-03-25T19:46:41.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.420+0000 7f29267fc640 1 -- 172.21.15.81:0/3082514548 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158574737 0 0) 0x7f29280b0a70 con 0x7f29280b1730 2024-03-25T19:46:41.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.420+0000 7f29267fc640 10 monclient(hunting): _negotiate 2024-03-25T19:46:41.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.420+0000 7f29267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:41.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.420+0000 7f29267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:41.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.420+0000 7f29267fc640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:46:41.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.420+0000 7f29267fc640 1 -- 172.21.15.81:0/3082514548 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29280af940 con 0x7f29280b1730 2024-03-25T19:46:41.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.421+0000 7f29267fc640 1 -- 172.21.15.81:0/3082514548 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3008076968 0 0) 0x7f2910003520 con 0x7f29280b4d60 2024-03-25T19:46:41.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.421+0000 7f29267fc640 1 -- 172.21.15.81:0/3082514548 >> v1:172.21.15.81:6790/0 conn(0x7f292814be90 legacy=0x7f292814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:41.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.421+0000 7f29267fc640 1 -- 172.21.15.81:0/3082514548 >> v1:172.21.15.98:6789/0 conn(0x7f29280b1730 legacy=0x7f29280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:41.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.421+0000 7f29267fc640 10 monclient: _finish_hunting 0 2024-03-25T19:46:41.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.421+0000 7f29267fc640 1 monclient: found mon.noname-a 2024-03-25T19:46:41.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.421+0000 7f29267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:41.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.421+0000 7f29267fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:46:41.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.421+0000 7f29267fc640 1 -- 172.21.15.81:0/3082514548 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29280b1030 con 0x7f29280b4d60 2024-03-25T19:46:41.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.421+0000 7f29267fc640 10 monclient: _finish_auth 0 2024-03-25T19:46:41.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.421+0000 7f29267fc640 10 monclient: _check_auth_tickets 2024-03-25T19:46:41.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.421+0000 7f29267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:41.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.421+0000 7f29267fc640 1 -- 172.21.15.81:0/3082514548 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f2918001690 con 0x7f29280b4d60 2024-03-25T19:46:41.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.421+0000 7f29267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:41.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.421+0000 7f29267fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:41.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.421+0000 7f29267fc640 10 monclient: dump: 2024-03-25T19:46:41.422 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:41.422 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:41.422 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:41.422 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:41.422 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:41.422 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:41.422 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:41.422 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:41.423 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:41.423 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:41.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.421+0000 7f292db5d640 5 monclient: authenticate success, global_id 4440 2024-03-25T19:46:41.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.421+0000 7f292db5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:41.430 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.431+0000 7f29267fc640 1 -- 172.21.15.81:0/3082514548 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29180017f0 con 0x7f29280b4d60 2024-03-25T19:46:41.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.431+0000 7f29267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:41.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.431+0000 7f29267fc640 1 -- 172.21.15.81:0/3082514548 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f2918001c00 con 0x7f29280b4d60 2024-03-25T19:46:41.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.431+0000 7f292db5d640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:41.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.431+0000 7f292db5d640 4 set_mon_vals no callback set 2024-03-25T19:46:41.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.431+0000 7f29267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:41.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.431+0000 7f29267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:41.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.431+0000 7f29267fc640 10 monclient: dump: 2024-03-25T19:46:41.431 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:41.431 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:41.431 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:41.431 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:41.431 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:41.431 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:41.432 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:41.432 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:41.432 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:41.432 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:41.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.432+0000 7f292db5d640 10 monclient: shutdown 2024-03-25T19:46:41.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.432+0000 7f292db5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:41.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.432+0000 7f292db5d640 1 -- 172.21.15.81:0/3082514548 >> v1:172.21.15.81:6789/0 conn(0x7f29280b4d60 legacy=0x7f292813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:41.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.432+0000 7f292db5d640 1 -- 172.21.15.81:0/3082514548 shutdown_connections 2024-03-25T19:46:41.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.432+0000 7f292db5d640 1 -- 172.21.15.81:0/3082514548 >> 172.21.15.81:0/3082514548 conn(0x7f29280617a0 msgr2=0x7f2928061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:41.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.432+0000 7f292db5d640 1 -- 172.21.15.81:0/3082514548 shutdown_connections 2024-03-25T19:46:41.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.432+0000 7f292db5d640 1 -- 172.21.15.81:0/3082514548 wait complete. 2024-03-25T19:46:41.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.433+0000 7f292db5d640 10 monclient: build_initial_monmap 2024-03-25T19:46:41.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.433+0000 7f292db5d640 10 monclient: monmap: 2024-03-25T19:46:41.435 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:46:41.435 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:41.435 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:46:41.435 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:46:41.435 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:41.435 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:46:41.435 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:46:41.435 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:46:41.435 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:46:41.435 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:41.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.433+0000 7f292db5d640 1 librados: starting msgr at 2024-03-25T19:46:41.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.433+0000 7f292db5d640 1 librados: starting objecter 2024-03-25T19:46:41.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.433+0000 7f292db5d640 1 Processor -- start 2024-03-25T19:46:41.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.433+0000 7f292db5d640 1 -- start start 2024-03-25T19:46:41.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.433+0000 7f292db5d640 1 librados: setting wanted keys 2024-03-25T19:46:41.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.433+0000 7f292db5d640 1 librados: calling monclient init 2024-03-25T19:46:41.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.433+0000 7f292db5d640 10 monclient: init 2024-03-25T19:46:41.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.434+0000 7f292db5d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:41.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.434+0000 7f292db5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:41.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.434+0000 7f292db5d640 10 monclient(hunting): picked mon.noname-c con 0x7f29280b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:46:41.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.434+0000 7f292db5d640 10 monclient(hunting): picked mon.noname-b con 0x7f29280b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:46:41.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.434+0000 7f292db5d640 10 monclient(hunting): picked mon.noname-a con 0x7f292814be90 addr v1:172.21.15.81:6789/0 2024-03-25T19:46:41.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.434+0000 7f292db5d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29281368f0 con 0x7f292814be90 2024-03-25T19:46:41.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.434+0000 7f292db5d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2928137750 con 0x7f29280b4d60 2024-03-25T19:46:41.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.434+0000 7f292db5d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29281363d0 con 0x7f29280b1730 2024-03-25T19:46:41.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.434+0000 7f292db5d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:41.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.434+0000 7f292db5d640 10 monclient(hunting): authenticate will time out at 4236.516103s 2024-03-25T19:46:41.444 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.445+0000 7f2927fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f292814be90 0x7f29281d7830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:55786/0 (socket says 172.21.15.81:55786) 2024-03-25T19:46:41.444 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.445+0000 7f2927fff640 1 -- 172.21.15.81:0/1505829653 learned_addr learned my addr 172.21.15.81:0/1505829653 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:46:41.445 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.446+0000 7f290ffff640 10 client.?.objecter ms_handle_connect 0x7f292814be90 2024-03-25T19:46:41.445 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.446+0000 7f290ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:41.447 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.448+0000 7f290ffff640 10 client.?.objecter ms_handle_connect 0x7f29280b4d60 2024-03-25T19:46:41.447 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.448+0000 7f290ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:41.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.450+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226432247 0 0) 0x7f29281368f0 con 0x7f292814be90 2024-03-25T19:46:41.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.451+0000 7f290ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:41.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.451+0000 7f290ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:41.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.451+0000 7f290ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:41.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.451+0000 7f290ffff640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:46:41.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.451+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2914004eb0 con 0x7f292814be90 2024-03-25T19:46:41.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.457+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159599803 0 0) 0x7f2928137750 con 0x7f29280b4d60 2024-03-25T19:46:41.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.457+0000 7f290ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:41.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.457+0000 7f290ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:41.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.457+0000 7f290ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:41.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.457+0000 7f290ffff640 10 monclient(hunting): my global_id is 4441 2024-03-25T19:46:41.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.457+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29281368f0 con 0x7f29280b4d60 2024-03-25T19:46:41.459 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.459+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1384176982 0 0) 0x7f2914004eb0 con 0x7f292814be90 2024-03-25T19:46:41.459 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.459+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2928137750 con 0x7f292814be90 2024-03-25T19:46:41.459 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.459+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f2920001c30 con 0x7f292814be90 2024-03-25T19:46:41.460 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.459+0000 7f290ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:41.460 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.459+0000 7f290ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:41.460 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.459+0000 7f290ffff640 10 monclient(hunting): dump: 2024-03-25T19:46:41.460 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:41.460 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:41.460 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:41.460 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:41.460 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:41.460 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:41.460 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:41.460 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:41.460 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:41.461 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:41.461 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.459+0000 7f290ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:41.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.468+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2365870405 0 0) 0x7f29281368f0 con 0x7f29280b4d60 2024-03-25T19:46:41.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.468+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2914004eb0 con 0x7f29280b4d60 2024-03-25T19:46:41.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.468+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f291c001930 con 0x7f29280b4d60 2024-03-25T19:46:41.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.468+0000 7f290ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:41.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.468+0000 7f290ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:41.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.468+0000 7f290ffff640 10 monclient(hunting): dump: 2024-03-25T19:46:41.468 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:41.468 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:41.468 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:41.468 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:41.468 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:41.468 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:41.469 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:41.469 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:41.469 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:41.469 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:41.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1120241689 0 0) 0x7f2928137750 con 0x7f292814be90 2024-03-25T19:46:41.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 >> v1:172.21.15.81:6790/0 conn(0x7f29280b1730 legacy=0x7f2928135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:41.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 >> v1:172.21.15.98:6789/0 conn(0x7f29280b4d60 legacy=0x7f292812fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:41.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f290ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:46:41.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f290ffff640 1 monclient: found mon.a 2024-03-25T19:46:41.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f290ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:41.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f290ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:41.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2928130470 con 0x7f292814be90 2024-03-25T19:46:41.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f290ffff640 10 monclient: _finish_auth 0 2024-03-25T19:46:41.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f290ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:46:41.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f290ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:41.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f292db5d640 5 monclient: authenticate success, global_id 4443 2024-03-25T19:46:41.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f292db5d640 10 monclient: _renew_subs 2024-03-25T19:46:41.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f292db5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:41.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f292db5d640 1 -- 172.21.15.81:0/1505829653 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2928130720 con 0x7f292814be90 2024-03-25T19:46:41.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f292db5d640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:41.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f292db5d640 10 monclient: _renew_subs 2024-03-25T19:46:41.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f292db5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:41.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f292db5d640 1 -- 172.21.15.81:0/1505829653 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29281322f0 con 0x7f292814be90 2024-03-25T19:46:41.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.474+0000 7f292db5d640 1 librados: init done 2024-03-25T19:46:41.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.475+0000 7f292db5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:41.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.475+0000 7f292db5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:41.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.475+0000 7f292db5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:41.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.475+0000 7f292db5d640 1 -- 172.21.15.81:0/1505829653 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2928136210 con 0x7f292814be90 2024-03-25T19:46:41.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.777+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2920003990 con 0x7f292814be90 2024-03-25T19:46:41.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.777+0000 7f290ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:41.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.777+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f2920003da0 con 0x7f292814be90 2024-03-25T19:46:41.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.777+0000 7f290ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:41.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.777+0000 7f290ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:41.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.777+0000 7f290ffff640 10 monclient: dump: 2024-03-25T19:46:41.777 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:41.777 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:41.777 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:41.778 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:41.778 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:41.778 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:41.778 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:41.778 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:41.778 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:41.778 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:41.778 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.777+0000 7f2925ffb640 4 set_mon_vals no callback set 2024-03-25T19:46:41.779 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.777+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f2920003fc0 con 0x7f292814be90 2024-03-25T19:46:41.779 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.778+0000 7f29277fe640 10 monclient: get_auth_request con 0x7f291406dec0 auth_method 0 2024-03-25T19:46:41.779 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.778+0000 7f290ffff640 10 client.4443.objecter ms_handle_connect 0x7f291406dec0 2024-03-25T19:46:41.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.783+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f29200549e0 con 0x7f292814be90 2024-03-25T19:46:41.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.783+0000 7f290ffff640 10 client.4443.objecter ms_dispatch 0x7f2928000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:41.783 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.783+0000 7f290ffff640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:41.783 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.783+0000 7f290ffff640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:41.783 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.783+0000 7f290ffff640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-25T19:46:41.783 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.783+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f29200504f0 con 0x7f292814be90 2024-03-25T19:46:41.783 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.783+0000 7f290ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:41.783 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:41.783+0000 7f290ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:42.031 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.031+0000 7f292db5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:42.031 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.031+0000 7f292db5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:42.031 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.031+0000 7f292db5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:42.031 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.031+0000 7f292db5d640 1 -- 172.21.15.81:0/1505829653 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29280b2270 con 0x7f292814be90 2024-03-25T19:46:42.057 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.057+0000 7f290ffff640 1 -- 172.21.15.81:0/1505829653 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f2920052420 con 0x7f292814be90 2024-03-25T19:46:42.057 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.057+0000 7f290ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:42.057 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.057+0000 7f290ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:42.057 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:46:42.057 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:42.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.059+0000 7f292db5d640 10 librados: watch_flush enter 2024-03-25T19:46:42.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.059+0000 7f292db5d640 10 librados: watch_flush exit 2024-03-25T19:46:42.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.059+0000 7f292db5d640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-25T19:46:42.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.059+0000 7f292db5d640 10 client.4443.objecter successfully canceled tick 2024-03-25T19:46:42.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.059+0000 7f292db5d640 1 -- 172.21.15.81:0/1505829653 >> v1:172.21.15.98:6800/3980829775 conn(0x7f291406dec0 legacy=0x7f2914070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:42.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.059+0000 7f292db5d640 10 monclient: shutdown 2024-03-25T19:46:42.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.059+0000 7f292db5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:42.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.059+0000 7f292db5d640 1 -- 172.21.15.81:0/1505829653 >> v1:172.21.15.81:6789/0 conn(0x7f292814be90 legacy=0x7f29281d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:42.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.059+0000 7f292db5d640 1 -- 172.21.15.81:0/1505829653 shutdown_connections 2024-03-25T19:46:42.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.059+0000 7f292db5d640 1 -- 172.21.15.81:0/1505829653 >> 172.21.15.81:0/1505829653 conn(0x7f29280617a0 msgr2=0x7f2928138500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:42.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.060+0000 7f292db5d640 1 -- 172.21.15.81:0/1505829653 shutdown_connections 2024-03-25T19:46:42.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.060+0000 7f292db5d640 1 -- 172.21.15.81:0/1505829653 wait complete. 2024-03-25T19:46:42.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:42.060+0000 7f292db5d640 1 librados: shutdown 2024-03-25T19:46:42.071 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:48.877 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:48.994 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:48.994+0000 7fa6508ba640 10 monclient: get_monmap_and_config 2024-03-25T19:46:48.994 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:48.994+0000 7fa6508ba640 10 monclient: build_initial_monmap 2024-03-25T19:46:48.994 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:48.994+0000 7fa6508ba640 10 monclient: monmap: 2024-03-25T19:46:48.995 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:46:48.995 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:48.995 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:46:48.995852+0000 2024-03-25T19:46:48.995 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:46:48.995852+0000 2024-03-25T19:46:48.995 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:48.995 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:48.995 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:46:48.995 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:46:48.995 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:46:48.995 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:48.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:48.996+0000 7fa6508ba640 1 Processor -- start 2024-03-25T19:46:48.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:48.996+0000 7fa6508ba640 1 -- start start 2024-03-25T19:46:48.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:48.996+0000 7fa6508ba640 10 monclient: init 2024-03-25T19:46:48.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:48.996+0000 7fa6508ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:48.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:48.996+0000 7fa6508ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:48.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:48.996+0000 7fa6508ba640 10 monclient(hunting): picked mon.noname-a con 0x7fa64c0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:46:48.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:48.996+0000 7fa6508ba640 10 monclient(hunting): picked mon.noname-b con 0x7fa64c13f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:46:48.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:48.997+0000 7fa6508ba640 10 monclient(hunting): picked mon.noname-c con 0x7fa64c1430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:46:48.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:48.997+0000 7fa6508ba640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c142520 con 0x7fa64c0a98f0 2024-03-25T19:46:48.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:48.997+0000 7fa6508ba640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c0b28a0 con 0x7fa64c1430c0 2024-03-25T19:46:48.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:48.997+0000 7fa6508ba640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c0b2a30 con 0x7fa64c13f710 2024-03-25T19:46:48.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:48.997+0000 7fa6508ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:48.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:48.997+0000 7fa6508ba640 10 monclient(hunting): authenticate will time out at 4244.078963s 2024-03-25T19:46:49.004 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.005+0000 7fa64ad9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa64c1430c0 0x7fa64c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:42180/0 (socket says 172.21.15.81:42180) 2024-03-25T19:46:49.004 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.005+0000 7fa64ad9d640 1 -- 172.21.15.81:0/1615035467 learned_addr learned my addr 172.21.15.81:0/1615035467 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:46:49.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.017+0000 7fa64959a640 1 -- 172.21.15.81:0/1615035467 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370349878 0 0) 0x7fa64c142520 con 0x7fa64c0a98f0 2024-03-25T19:46:49.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.017+0000 7fa64959a640 10 monclient(hunting): _negotiate 2024-03-25T19:46:49.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.017+0000 7fa64959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:49.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.017+0000 7fa64959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:49.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.017+0000 7fa64959a640 10 monclient(hunting): my global_id is 4446 2024-03-25T19:46:49.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.017+0000 7fa64959a640 1 -- 172.21.15.81:0/1615035467 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa634003520 con 0x7fa64c0a98f0 2024-03-25T19:46:49.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.017+0000 7fa64959a640 1 -- 172.21.15.81:0/1615035467 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754474333 0 0) 0x7fa64c0b28a0 con 0x7fa64c1430c0 2024-03-25T19:46:49.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.017+0000 7fa64959a640 10 monclient(hunting): _negotiate 2024-03-25T19:46:49.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.017+0000 7fa64959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:49.018 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.017+0000 7fa64959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:49.018 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.017+0000 7fa64959a640 10 monclient(hunting): my global_id is 4444 2024-03-25T19:46:49.018 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.017+0000 7fa64959a640 1 -- 172.21.15.81:0/1615035467 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa64c142520 con 0x7fa64c1430c0 2024-03-25T19:46:49.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.033+0000 7fa64959a640 1 -- 172.21.15.81:0/1615035467 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2193364483 0 0) 0x7fa64c142520 con 0x7fa64c1430c0 2024-03-25T19:46:49.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.033+0000 7fa64959a640 1 -- 172.21.15.81:0/1615035467 >> v1:172.21.15.81:6790/0 conn(0x7fa64c13f710 legacy=0x7fa64c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:49.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.033+0000 7fa64959a640 1 -- 172.21.15.81:0/1615035467 >> v1:172.21.15.81:6789/0 conn(0x7fa64c0a98f0 legacy=0x7fa64c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:49.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.033+0000 7fa64959a640 10 monclient: _finish_hunting 0 2024-03-25T19:46:49.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.033+0000 7fa64959a640 1 monclient: found mon.noname-c 2024-03-25T19:46:49.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.033+0000 7fa64959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:49.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.033+0000 7fa64959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:46:49.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.033+0000 7fa64959a640 1 -- 172.21.15.81:0/1615035467 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c0b44c0 con 0x7fa64c1430c0 2024-03-25T19:46:49.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.033+0000 7fa64959a640 10 monclient: _finish_auth 0 2024-03-25T19:46:49.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.033+0000 7fa64959a640 10 monclient: _check_auth_tickets 2024-03-25T19:46:49.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.033+0000 7fa64959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:49.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.033+0000 7fa64959a640 1 -- 172.21.15.81:0/1615035467 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa6440019f0 con 0x7fa64c1430c0 2024-03-25T19:46:49.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.033+0000 7fa64959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:49.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.033+0000 7fa64959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:49.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.033+0000 7fa64959a640 10 monclient: dump: 2024-03-25T19:46:49.035 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:49.035 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:49.035 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:49.035 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:49.035 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:49.035 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:49.035 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:49.035 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:49.035 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:49.035 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:49.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.033+0000 7fa6508ba640 5 monclient: authenticate success, global_id 4444 2024-03-25T19:46:49.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.033+0000 7fa6508ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:49.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.052+0000 7fa64959a640 1 -- 172.21.15.81:0/1615035467 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa644001b50 con 0x7fa64c1430c0 2024-03-25T19:46:49.052 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.052+0000 7fa64959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:49.052 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.052+0000 7fa64959a640 1 -- 172.21.15.81:0/1615035467 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa644001f20 con 0x7fa64c1430c0 2024-03-25T19:46:49.052 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.052+0000 7fa64959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:49.052 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.052+0000 7fa64959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:49.052 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.052+0000 7fa64959a640 10 monclient: dump: 2024-03-25T19:46:49.052 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:49.052 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:49.052 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:49.052 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:49.052 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:49.052 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:49.052 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:49.052 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:49.053 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:49.053 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:49.053 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.052+0000 7fa6508ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:49.053 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.052+0000 7fa6508ba640 4 set_mon_vals no callback set 2024-03-25T19:46:49.054 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.053+0000 7fa6508ba640 10 monclient: shutdown 2024-03-25T19:46:49.054 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.053+0000 7fa6508ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:49.054 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.053+0000 7fa6508ba640 1 -- 172.21.15.81:0/1615035467 >> v1:172.21.15.98:6789/0 conn(0x7fa64c1430c0 legacy=0x7fa64c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:49.054 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.053+0000 7fa6508ba640 1 -- 172.21.15.81:0/1615035467 shutdown_connections 2024-03-25T19:46:49.054 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.053+0000 7fa6508ba640 1 -- 172.21.15.81:0/1615035467 >> 172.21.15.81:0/1615035467 conn(0x7fa64c0b7ea0 msgr2=0x7fa64c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:49.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.053+0000 7fa6508ba640 1 -- 172.21.15.81:0/1615035467 shutdown_connections 2024-03-25T19:46:49.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.053+0000 7fa6508ba640 1 -- 172.21.15.81:0/1615035467 wait complete. 2024-03-25T19:46:49.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.054+0000 7fa6508ba640 10 monclient: build_initial_monmap 2024-03-25T19:46:49.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.054+0000 7fa6508ba640 10 monclient: monmap: 2024-03-25T19:46:49.055 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:46:49.055 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:49.055 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:46:49.055 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:46:49.055 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:49.055 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:46:49.055 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:46:49.055 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:46:49.055 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:46:49.056 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:49.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.054+0000 7fa6508ba640 1 librados: starting msgr at 2024-03-25T19:46:49.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.054+0000 7fa6508ba640 1 librados: starting objecter 2024-03-25T19:46:49.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.054+0000 7fa6508ba640 1 Processor -- start 2024-03-25T19:46:49.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.054+0000 7fa6508ba640 1 -- start start 2024-03-25T19:46:49.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.054+0000 7fa6508ba640 1 librados: setting wanted keys 2024-03-25T19:46:49.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.054+0000 7fa6508ba640 1 librados: calling monclient init 2024-03-25T19:46:49.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.054+0000 7fa6508ba640 10 monclient: init 2024-03-25T19:46:49.057 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.055+0000 7fa6508ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:49.057 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.055+0000 7fa6508ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:49.057 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.055+0000 7fa6508ba640 10 monclient(hunting): picked mon.noname-a con 0x7fa64c0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:46:49.057 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.055+0000 7fa6508ba640 10 monclient(hunting): picked mon.noname-c con 0x7fa64c13f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:46:49.057 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.055+0000 7fa6508ba640 10 monclient(hunting): picked mon.noname-b con 0x7fa64c1430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:46:49.057 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.055+0000 7fa6508ba640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c0ad410 con 0x7fa64c0a98f0 2024-03-25T19:46:49.057 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.055+0000 7fa6508ba640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c1d7d80 con 0x7fa64c1430c0 2024-03-25T19:46:49.058 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.055+0000 7fa6508ba640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa64c1d8360 con 0x7fa64c13f710 2024-03-25T19:46:49.058 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.055+0000 7fa6508ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:49.058 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.055+0000 7fa6508ba640 10 monclient(hunting): authenticate will time out at 4244.137706s 2024-03-25T19:46:49.062 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.063+0000 7fa64a59c640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fa64c0a98f0 0x7fa64c0b10c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:38570/0 (socket says 172.21.15.81:38570) 2024-03-25T19:46:49.062 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.063+0000 7fa64a59c640 1 -- 172.21.15.81:0/3957571270 learned_addr learned my addr 172.21.15.81:0/3957571270 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:46:49.065 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.065+0000 7fa63affd640 10 client.?.objecter ms_handle_connect 0x7fa64c0a98f0 2024-03-25T19:46:49.065 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.065+0000 7fa63affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:49.068 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.069+0000 7fa63affd640 10 client.?.objecter ms_handle_connect 0x7fa64c13f710 2024-03-25T19:46:49.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.069+0000 7fa63affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:49.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.073+0000 7fa63affd640 10 client.?.objecter ms_handle_connect 0x7fa64c1430c0 2024-03-25T19:46:49.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.073+0000 7fa63affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:49.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.078+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044425112 0 0) 0x7fa64c1d8360 con 0x7fa64c13f710 2024-03-25T19:46:49.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.078+0000 7fa63affd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:49.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.078+0000 7fa63affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:49.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.078+0000 7fa63affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:49.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.078+0000 7fa63affd640 10 monclient(hunting): my global_id is 4427 2024-03-25T19:46:49.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.078+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa62c004eb0 con 0x7fa64c13f710 2024-03-25T19:46:49.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.080+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200525614 0 0) 0x7fa64c0ad410 con 0x7fa64c0a98f0 2024-03-25T19:46:49.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.080+0000 7fa63affd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:49.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.080+0000 7fa63affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:49.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.080+0000 7fa63affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:49.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.080+0000 7fa63affd640 10 monclient(hunting): my global_id is 4449 2024-03-25T19:46:49.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.080+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa64c1d8360 con 0x7fa64c0a98f0 2024-03-25T19:46:49.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.095+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3242798877 0 0) 0x7fa64c1d7d80 con 0x7fa64c1430c0 2024-03-25T19:46:49.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.095+0000 7fa63affd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:49.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.095+0000 7fa63affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:49.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.095+0000 7fa63affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:49.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.095+0000 7fa63affd640 10 monclient(hunting): my global_id is 4447 2024-03-25T19:46:49.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.095+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa64c0ad410 con 0x7fa64c1430c0 2024-03-25T19:46:49.100 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.100+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1992657981 0 0) 0x7fa64c1d8360 con 0x7fa64c0a98f0 2024-03-25T19:46:49.100 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.100+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa64c1d7d80 con 0x7fa64c0a98f0 2024-03-25T19:46:49.100 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.100+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa63c001990 con 0x7fa64c0a98f0 2024-03-25T19:46:49.100 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.100+0000 7fa63affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:49.101 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.100+0000 7fa63affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:49.101 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.100+0000 7fa63affd640 10 monclient(hunting): dump: 2024-03-25T19:46:49.101 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:49.101 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:49.101 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:49.101 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:49.101 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:49.101 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:49.101 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:49.101 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:49.101 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:49.101 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:49.101 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.100+0000 7fa63affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:49.106 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.107+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4102322752 0 0) 0x7fa62c004eb0 con 0x7fa64c13f710 2024-03-25T19:46:49.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.107+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa64c1d8360 con 0x7fa64c13f710 2024-03-25T19:46:49.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.107+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa6400018c0 con 0x7fa64c13f710 2024-03-25T19:46:49.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.107+0000 7fa63affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:49.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.107+0000 7fa63affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:49.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.107+0000 7fa63affd640 10 monclient(hunting): dump: 2024-03-25T19:46:49.107 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:49.107 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:49.107 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:49.107 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:49.107 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:49.107 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:49.107 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:49.107 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:49.107 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:49.107 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:49.120 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2520598559 0 0) 0x7fa64c1d7d80 con 0x7fa64c0a98f0 2024-03-25T19:46:49.120 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 >> v1:172.21.15.81:6790/0 conn(0x7fa64c13f710 legacy=0x7fa64c0acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:49.120 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 >> v1:172.21.15.98:6789/0 conn(0x7fa64c1430c0 legacy=0x7fa64c1d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:49.120 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa63affd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:49.120 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa63affd640 1 monclient: found mon.a 2024-03-25T19:46:49.120 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa63affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:49.120 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa63affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:49.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64c1d84f0 con 0x7fa64c0a98f0 2024-03-25T19:46:49.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa63affd640 10 monclient: _finish_auth 0 2024-03-25T19:46:49.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa63affd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:49.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa63affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:49.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa6508ba640 5 monclient: authenticate success, global_id 4449 2024-03-25T19:46:49.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa6508ba640 10 monclient: _renew_subs 2024-03-25T19:46:49.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa6508ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:49.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa6508ba640 1 -- 172.21.15.81:0/3957571270 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64c14e560 con 0x7fa64c0a98f0 2024-03-25T19:46:49.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa6508ba640 10 client.4449.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:49.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa6508ba640 10 monclient: _renew_subs 2024-03-25T19:46:49.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa6508ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:49.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa6508ba640 1 -- 172.21.15.81:0/3957571270 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64c1d41e0 con 0x7fa64c0a98f0 2024-03-25T19:46:49.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.120+0000 7fa6508ba640 1 librados: init done 2024-03-25T19:46:49.122 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.121+0000 7fa6508ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:49.122 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.121+0000 7fa6508ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:49.122 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.121+0000 7fa6508ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:49.122 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.121+0000 7fa6508ba640 1 -- 172.21.15.81:0/3957571270 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64c0b3ef0 con 0x7fa64c0a98f0 2024-03-25T19:46:49.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.450+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa63c003610 con 0x7fa64c0a98f0 2024-03-25T19:46:49.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.450+0000 7fa63affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:49.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.450+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa63c0039e0 con 0x7fa64c0a98f0 2024-03-25T19:46:49.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.450+0000 7fa63affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:49.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.450+0000 7fa63affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:49.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.450+0000 7fa63affd640 10 monclient: dump: 2024-03-25T19:46:49.450 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:49.450 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:49.450 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:49.450 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:49.451 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:49.451 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:49.451 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:49.451 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:49.451 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:49.451 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:49.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.450+0000 7fa648d99640 4 set_mon_vals no callback set 2024-03-25T19:46:49.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.451+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fa63c003c00 con 0x7fa64c0a98f0 2024-03-25T19:46:49.452 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.452+0000 7fa649d9b640 10 monclient: get_auth_request con 0x7fa62c06df70 auth_method 0 2024-03-25T19:46:49.452 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.452+0000 7fa63affd640 10 client.4449.objecter ms_handle_connect 0x7fa62c06df70 2024-03-25T19:46:49.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.456+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fa63c054800 con 0x7fa64c0a98f0 2024-03-25T19:46:49.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.456+0000 7fa63affd640 10 client.4449.objecter ms_dispatch 0x7fa64c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:49.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.456+0000 7fa63affd640 3 client.4449.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:49.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.456+0000 7fa63affd640 3 client.4449.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:49.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.456+0000 7fa63affd640 20 client.4449.objecter dump_active .. 0 homeless 2024-03-25T19:46:49.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.456+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa63c050310 con 0x7fa64c0a98f0 2024-03-25T19:46:49.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.456+0000 7fa63affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:49.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.456+0000 7fa63affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:49.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.676+0000 7fa6508ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:49.676 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.676+0000 7fa6508ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:49.676 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.676+0000 7fa6508ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:49.676 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.676+0000 7fa6508ba640 1 -- 172.21.15.81:0/3957571270 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa64c0627b0 con 0x7fa64c0a98f0 2024-03-25T19:46:49.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.761+0000 7fa63affd640 1 -- 172.21.15.81:0/3957571270 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fa63c052240 con 0x7fa64c0a98f0 2024-03-25T19:46:49.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.761+0000 7fa63affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:49.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.761+0000 7fa63affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:49.761 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:46:49.761 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:49.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.764+0000 7fa6508ba640 10 librados: watch_flush enter 2024-03-25T19:46:49.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.764+0000 7fa6508ba640 10 librados: watch_flush exit 2024-03-25T19:46:49.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.764+0000 7fa6508ba640 20 client.4449.objecter shutdown clearing up homeless session... 2024-03-25T19:46:49.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.764+0000 7fa6508ba640 10 client.4449.objecter successfully canceled tick 2024-03-25T19:46:49.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.764+0000 7fa6508ba640 1 -- 172.21.15.81:0/3957571270 >> v1:172.21.15.98:6800/3980829775 conn(0x7fa62c06df70 legacy=0x7fa62c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:49.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.764+0000 7fa6508ba640 10 monclient: shutdown 2024-03-25T19:46:49.767 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.764+0000 7fa6508ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:49.767 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.764+0000 7fa6508ba640 1 -- 172.21.15.81:0/3957571270 >> v1:172.21.15.81:6789/0 conn(0x7fa64c0a98f0 legacy=0x7fa64c0b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:49.767 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.764+0000 7fa6508ba640 1 -- 172.21.15.81:0/3957571270 shutdown_connections 2024-03-25T19:46:49.767 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.764+0000 7fa6508ba640 1 -- 172.21.15.81:0/3957571270 >> 172.21.15.81:0/3957571270 conn(0x7fa64c0b7ea0 msgr2=0x7fa64c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:49.767 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.764+0000 7fa6508ba640 1 -- 172.21.15.81:0/3957571270 shutdown_connections 2024-03-25T19:46:49.767 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.764+0000 7fa6508ba640 1 -- 172.21.15.81:0/3957571270 wait complete. 2024-03-25T19:46:49.767 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:49.764+0000 7fa6508ba640 1 librados: shutdown 2024-03-25T19:46:49.778 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:56.585 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:56.698 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.698+0000 7face6ba8640 10 monclient: get_monmap_and_config 2024-03-25T19:46:56.698 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.698+0000 7face6ba8640 10 monclient: build_initial_monmap 2024-03-25T19:46:56.699 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.698+0000 7face6ba8640 10 monclient: monmap: 2024-03-25T19:46:56.699 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:46:56.699 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:56.699 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:46:56.700030+0000 2024-03-25T19:46:56.699 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:46:56.700030+0000 2024-03-25T19:46:56.699 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:56.699 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:56.699 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:46:56.699 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:46:56.699 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:46:56.699 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:56.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.699+0000 7face6ba8640 1 Processor -- start 2024-03-25T19:46:56.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.699+0000 7face6ba8640 1 -- start start 2024-03-25T19:46:56.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.699+0000 7face6ba8640 10 monclient: init 2024-03-25T19:46:56.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.699+0000 7face6ba8640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:56.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.699+0000 7face6ba8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:56.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.699+0000 7face6ba8640 10 monclient(hunting): picked mon.noname-a con 0x7face00a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:46:56.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.699+0000 7face6ba8640 10 monclient(hunting): picked mon.noname-c con 0x7face013f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:46:56.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.699+0000 7face6ba8640 10 monclient(hunting): picked mon.noname-b con 0x7face01430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:46:56.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.699+0000 7face6ba8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face0142520 con 0x7face00a98f0 2024-03-25T19:46:56.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.699+0000 7face6ba8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face00b28a0 con 0x7face013f710 2024-03-25T19:46:56.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.699+0000 7face6ba8640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face00b2a30 con 0x7face01430c0 2024-03-25T19:46:56.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.699+0000 7face6ba8640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:56.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.699+0000 7face6ba8640 10 monclient(hunting): authenticate will time out at 4251.782800s 2024-03-25T19:46:56.713 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.713+0000 7facdffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7face013f710 0x7face01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:47368/0 (socket says 172.21.15.81:47368) 2024-03-25T19:46:56.713 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.713+0000 7facdffff640 1 -- 172.21.15.81:0/3780668855 learned_addr learned my addr 172.21.15.81:0/3780668855 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:46:56.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.728+0000 7facdf7fe640 1 -- 172.21.15.81:0/3780668855 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80861254 0 0) 0x7face00b28a0 con 0x7face013f710 2024-03-25T19:46:56.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.728+0000 7facdf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:56.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.728+0000 7facdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:56.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.728+0000 7facdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:56.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.728+0000 7facdf7fe640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:46:56.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.728+0000 7facdf7fe640 1 -- 172.21.15.81:0/3780668855 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facd0003520 con 0x7face013f710 2024-03-25T19:46:56.730 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.729+0000 7facdf7fe640 1 -- 172.21.15.81:0/3780668855 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272201329 0 0) 0x7face0142520 con 0x7face00a98f0 2024-03-25T19:46:56.730 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.729+0000 7facdf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:56.730 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.729+0000 7facdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:56.730 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.729+0000 7facdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:56.730 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.729+0000 7facdf7fe640 10 monclient(hunting): my global_id is 4452 2024-03-25T19:46:56.730 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.729+0000 7facdf7fe640 1 -- 172.21.15.81:0/3780668855 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face00b28a0 con 0x7face00a98f0 2024-03-25T19:46:56.737 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.737+0000 7facdf7fe640 1 -- 172.21.15.81:0/3780668855 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044158250 0 0) 0x7face00b2a30 con 0x7face01430c0 2024-03-25T19:46:56.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.737+0000 7facdf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:56.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.737+0000 7facdf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:56.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.737+0000 7facdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:56.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.737+0000 7facdf7fe640 10 monclient(hunting): my global_id is 4430 2024-03-25T19:46:56.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.737+0000 7facdf7fe640 1 -- 172.21.15.81:0/3780668855 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face0142520 con 0x7face01430c0 2024-03-25T19:46:56.740 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.739+0000 7facdf7fe640 1 -- 172.21.15.81:0/3780668855 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2779015572 0 0) 0x7facd0003520 con 0x7face013f710 2024-03-25T19:46:56.740 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.739+0000 7facdf7fe640 1 -- 172.21.15.81:0/3780668855 >> v1:172.21.15.81:6790/0 conn(0x7face01430c0 legacy=0x7face014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:56.740 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.739+0000 7facdf7fe640 1 -- 172.21.15.81:0/3780668855 >> v1:172.21.15.81:6789/0 conn(0x7face00a98f0 legacy=0x7face00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:56.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.739+0000 7facdf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:56.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.739+0000 7facdf7fe640 1 monclient: found mon.noname-c 2024-03-25T19:46:56.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.739+0000 7facdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:56.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.739+0000 7facdf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:46:56.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.739+0000 7facdf7fe640 1 -- 172.21.15.81:0/3780668855 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face00b44c0 con 0x7face013f710 2024-03-25T19:46:56.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.739+0000 7facdf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:56.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.739+0000 7facdf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:56.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.739+0000 7facdf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:56.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.739+0000 7facdf7fe640 1 -- 172.21.15.81:0/3780668855 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7faccc001690 con 0x7face013f710 2024-03-25T19:46:56.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.739+0000 7facdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:56.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.739+0000 7facdf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:56.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.739+0000 7facdf7fe640 10 monclient: dump: 2024-03-25T19:46:56.741 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:56.742 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:56.742 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:56.742 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:56.742 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:56.742 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:56.742 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:56.742 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:56.742 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:56.742 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:56.742 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.739+0000 7face6ba8640 5 monclient: authenticate success, global_id 4450 2024-03-25T19:46:56.742 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.739+0000 7face6ba8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:56.743 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.743+0000 7facdf7fe640 1 -- 172.21.15.81:0/3780668855 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faccc0017f0 con 0x7face013f710 2024-03-25T19:46:56.744 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.743+0000 7facdf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:56.744 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.743+0000 7facdf7fe640 1 -- 172.21.15.81:0/3780668855 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7faccc001c00 con 0x7face013f710 2024-03-25T19:46:56.744 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.743+0000 7facdf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:56.744 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.743+0000 7facdf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:56.744 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.743+0000 7facdf7fe640 10 monclient: dump: 2024-03-25T19:46:56.744 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:56.744 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:56.744 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:56.744 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:56.744 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:56.744 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:56.744 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:56.745 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:56.745 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:56.745 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:56.745 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.743+0000 7face6ba8640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:56.745 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.743+0000 7face6ba8640 4 set_mon_vals no callback set 2024-03-25T19:46:56.747 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.744+0000 7face6ba8640 10 monclient: shutdown 2024-03-25T19:46:56.747 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.744+0000 7face6ba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:56.747 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.744+0000 7face6ba8640 1 -- 172.21.15.81:0/3780668855 >> v1:172.21.15.98:6789/0 conn(0x7face013f710 legacy=0x7face01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:56.747 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.744+0000 7face6ba8640 1 -- 172.21.15.81:0/3780668855 shutdown_connections 2024-03-25T19:46:56.747 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.744+0000 7face6ba8640 1 -- 172.21.15.81:0/3780668855 >> 172.21.15.81:0/3780668855 conn(0x7face00b7ea0 msgr2=0x7face00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:56.747 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.744+0000 7face6ba8640 1 -- 172.21.15.81:0/3780668855 shutdown_connections 2024-03-25T19:46:56.747 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.744+0000 7face6ba8640 1 -- 172.21.15.81:0/3780668855 wait complete. 2024-03-25T19:46:56.748 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.744+0000 7face6ba8640 10 monclient: build_initial_monmap 2024-03-25T19:46:56.748 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.744+0000 7face6ba8640 10 monclient: monmap: 2024-03-25T19:46:56.748 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:46:56.748 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:56.748 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:46:56.748 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:46:56.748 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:56.748 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:46:56.748 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:46:56.748 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:46:56.748 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:46:56.748 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:56.748 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.745+0000 7face6ba8640 1 librados: starting msgr at 2024-03-25T19:46:56.748 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.745+0000 7face6ba8640 1 librados: starting objecter 2024-03-25T19:46:56.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.745+0000 7face6ba8640 1 Processor -- start 2024-03-25T19:46:56.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.745+0000 7face6ba8640 1 -- start start 2024-03-25T19:46:56.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.745+0000 7face6ba8640 1 librados: setting wanted keys 2024-03-25T19:46:56.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.745+0000 7face6ba8640 1 librados: calling monclient init 2024-03-25T19:46:56.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.745+0000 7face6ba8640 10 monclient: init 2024-03-25T19:46:56.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.745+0000 7face6ba8640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:56.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.745+0000 7face6ba8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:56.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.745+0000 7face6ba8640 10 monclient(hunting): picked mon.noname-b con 0x7face00a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:46:56.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.745+0000 7face6ba8640 10 monclient(hunting): picked mon.noname-a con 0x7face013f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:46:56.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.745+0000 7face6ba8640 10 monclient(hunting): picked mon.noname-c con 0x7face01430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:46:56.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.745+0000 7face6ba8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face01354c0 con 0x7face013f710 2024-03-25T19:46:56.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.746+0000 7face6ba8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face01319b0 con 0x7face00a98f0 2024-03-25T19:46:56.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.746+0000 7face6ba8640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face0135070 con 0x7face01430c0 2024-03-25T19:46:56.750 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.746+0000 7face6ba8640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:56.750 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.746+0000 7face6ba8640 10 monclient(hunting): authenticate will time out at 4251.828909s 2024-03-25T19:46:56.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.757+0000 7facdffff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7face013f710 0x7face0130b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:43610/0 (socket says 172.21.15.81:43610) 2024-03-25T19:46:56.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.757+0000 7facdffff640 1 -- 172.21.15.81:0/2159864112 learned_addr learned my addr 172.21.15.81:0/2159864112 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:46:56.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.759+0000 7facdd7fa640 10 client.?.objecter ms_handle_connect 0x7face013f710 2024-03-25T19:46:56.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.759+0000 7facdd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:56.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.766+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012503270 0 0) 0x7face01354c0 con 0x7face013f710 2024-03-25T19:46:56.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.766+0000 7facdd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:56.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.766+0000 7facdd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:56.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.766+0000 7facdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:56.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.766+0000 7facdd7fa640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:46:56.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.766+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facc4004eb0 con 0x7face013f710 2024-03-25T19:46:56.771 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.771+0000 7facdd7fa640 10 client.?.objecter ms_handle_connect 0x7face00a98f0 2024-03-25T19:46:56.771 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.771+0000 7facdd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:56.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.773+0000 7facdd7fa640 10 client.?.objecter ms_handle_connect 0x7face01430c0 2024-03-25T19:46:56.773 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.773+0000 7facdd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:56.780 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.781+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156956755 0 0) 0x7face01319b0 con 0x7face00a98f0 2024-03-25T19:46:56.780 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.781+0000 7facdd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:56.780 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.781+0000 7facdd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:56.780 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.781+0000 7facdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:56.781 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.781+0000 7facdd7fa640 10 monclient(hunting): my global_id is 4453 2024-03-25T19:46:56.781 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.781+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face01354c0 con 0x7face00a98f0 2024-03-25T19:46:56.783 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.783+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2575651090 0 0) 0x7facc4004eb0 con 0x7face013f710 2024-03-25T19:46:56.783 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.783+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7face01319b0 con 0x7face013f710 2024-03-25T19:46:56.783 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.783+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7faccc001bc0 con 0x7face013f710 2024-03-25T19:46:56.784 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.783+0000 7facdd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:56.784 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.783+0000 7facdd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:56.784 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.783+0000 7facdd7fa640 10 monclient(hunting): dump: 2024-03-25T19:46:56.784 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:56.784 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:56.784 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:56.784 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:56.784 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:56.784 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:56.784 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:56.784 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:56.784 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:56.784 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:56.784 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.783+0000 7facdd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:56.787 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.787+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189801377 0 0) 0x7face0135070 con 0x7face01430c0 2024-03-25T19:46:56.787 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.787+0000 7facdd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:56.787 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.787+0000 7facdd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:56.787 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.787+0000 7facdd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:56.787 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.787+0000 7facdd7fa640 10 monclient(hunting): my global_id is 4433 2024-03-25T19:46:56.787 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.787+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facc4004eb0 con 0x7face01430c0 2024-03-25T19:46:56.789 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.788+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1874425707 0 0) 0x7face01354c0 con 0x7face00a98f0 2024-03-25T19:46:56.789 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.788+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7face0135070 con 0x7face00a98f0 2024-03-25T19:46:56.789 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7facd4001990 con 0x7face00a98f0 2024-03-25T19:46:56.789 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7facdd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:56.789 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7facdd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:56.789 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7facdd7fa640 10 monclient(hunting): dump: 2024-03-25T19:46:56.790 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:56.790 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:56.790 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:56.790 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:56.790 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:56.790 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:56.790 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:56.790 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:56.790 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:56.790 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:56.790 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 951103473 0 0) 0x7face01319b0 con 0x7face013f710 2024-03-25T19:46:56.790 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 >> v1:172.21.15.81:6790/0 conn(0x7face01430c0 legacy=0x7face01312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:56.791 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 >> v1:172.21.15.98:6789/0 conn(0x7face00a98f0 legacy=0x7face01d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:56.791 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7facdd7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:46:56.791 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7facdd7fa640 1 monclient: found mon.a 2024-03-25T19:46:56.791 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7facdd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:56.791 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7facdd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:56.791 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face0135200 con 0x7face013f710 2024-03-25T19:46:56.791 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7facdd7fa640 10 monclient: _finish_auth 0 2024-03-25T19:46:56.791 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7facdd7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:46:56.791 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7facdd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:56.791 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7face6ba8640 5 monclient: authenticate success, global_id 4455 2024-03-25T19:46:56.791 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7face6ba8640 10 monclient: _renew_subs 2024-03-25T19:46:56.791 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7face6ba8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:56.791 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7face6ba8640 1 -- 172.21.15.81:0/2159864112 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face00b7010 con 0x7face013f710 2024-03-25T19:46:56.791 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7face6ba8640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:56.791 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7face6ba8640 10 monclient: _renew_subs 2024-03-25T19:46:56.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7face6ba8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:56.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7face6ba8640 1 -- 172.21.15.81:0/2159864112 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7face013b890 con 0x7face013f710 2024-03-25T19:46:56.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.789+0000 7face6ba8640 1 librados: init done 2024-03-25T19:46:56.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.791+0000 7face6ba8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:56.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.791+0000 7face6ba8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:56.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.791+0000 7face6ba8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:56.793 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:56.791+0000 7face6ba8640 1 -- 172.21.15.81:0/2159864112 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face0137160 con 0x7face013f710 2024-03-25T19:46:57.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.082+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faccc0011f0 con 0x7face013f710 2024-03-25T19:46:57.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.082+0000 7facdd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:57.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.082+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7faccc004cb0 con 0x7face013f710 2024-03-25T19:46:57.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.082+0000 7facdd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:57.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.082+0000 7facdd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:57.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.082+0000 7facdd7fa640 10 monclient: dump: 2024-03-25T19:46:57.082 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:46:57.082 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:46:57.083 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:57.083 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:46:57.083 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:57.083 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:46:57.083 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:46:57.083 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:46:57.083 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:46:57.083 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:46:57.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.082+0000 7facdeffd640 4 set_mon_vals no callback set 2024-03-25T19:46:57.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.083+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7faccc01ad00 con 0x7face013f710 2024-03-25T19:46:57.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.083+0000 7face4944640 10 monclient: get_auth_request con 0x7facc406de80 auth_method 0 2024-03-25T19:46:57.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.084+0000 7facdd7fa640 10 client.4455.objecter ms_handle_connect 0x7facc406de80 2024-03-25T19:46:57.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.089+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7faccc054990 con 0x7face013f710 2024-03-25T19:46:57.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.089+0000 7facdd7fa640 10 client.4455.objecter ms_dispatch 0x7face0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:57.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.089+0000 7facdd7fa640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:57.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.089+0000 7facdd7fa640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:57.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.089+0000 7facdd7fa640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-25T19:46:57.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.089+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faccc0502d0 con 0x7face013f710 2024-03-25T19:46:57.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.089+0000 7facdd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:57.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.089+0000 7facdd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:57.338 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.338+0000 7face6ba8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:57.338 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.338+0000 7face6ba8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:57.338 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.338+0000 7face6ba8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:46:57.338 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.338+0000 7face6ba8640 1 -- 172.21.15.81:0/2159864112 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7face00627b0 con 0x7face013f710 2024-03-25T19:46:57.371 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.371+0000 7facdd7fa640 1 -- 172.21.15.81:0/2159864112 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7faccc0523d0 con 0x7face013f710 2024-03-25T19:46:57.371 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.372+0000 7facdd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:57.371 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.372+0000 7facdd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:57.372 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:46:57.372 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:57.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.375+0000 7face6ba8640 10 librados: watch_flush enter 2024-03-25T19:46:57.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.375+0000 7face6ba8640 10 librados: watch_flush exit 2024-03-25T19:46:57.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.375+0000 7face6ba8640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-25T19:46:57.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.375+0000 7face6ba8640 10 client.4455.objecter successfully canceled tick 2024-03-25T19:46:57.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.375+0000 7face6ba8640 1 -- 172.21.15.81:0/2159864112 >> v1:172.21.15.98:6800/3980829775 conn(0x7facc406de80 legacy=0x7facc4070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:57.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.375+0000 7face6ba8640 10 monclient: shutdown 2024-03-25T19:46:57.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.375+0000 7face6ba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:57.376 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.375+0000 7face6ba8640 1 -- 172.21.15.81:0/2159864112 >> v1:172.21.15.81:6789/0 conn(0x7face013f710 legacy=0x7face0130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:57.376 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.375+0000 7face6ba8640 1 -- 172.21.15.81:0/2159864112 shutdown_connections 2024-03-25T19:46:57.376 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.375+0000 7face6ba8640 1 -- 172.21.15.81:0/2159864112 >> 172.21.15.81:0/2159864112 conn(0x7face00b7ea0 msgr2=0x7face013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:57.376 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.375+0000 7face6ba8640 1 -- 172.21.15.81:0/2159864112 shutdown_connections 2024-03-25T19:46:57.376 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.375+0000 7face6ba8640 1 -- 172.21.15.81:0/2159864112 wait complete. 2024-03-25T19:46:57.376 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:46:57.375+0000 7face6ba8640 1 librados: shutdown 2024-03-25T19:46:57.387 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:57.405 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:05:20.047 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:57.458 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:05:20.100 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:57.491 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:05:20.133 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:57.504 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:05:20.146 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:57.508 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:20.143 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:57.517 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:05:20.159 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:57.518 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:20.152 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:57.537 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:20.172 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:57.544 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:05:20.186 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:04.193 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:04.310 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.311+0000 7ff51adc7640 10 monclient: get_monmap_and_config 2024-03-25T19:47:04.310 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.311+0000 7ff51adc7640 10 monclient: build_initial_monmap 2024-03-25T19:47:04.311 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.311+0000 7ff51adc7640 10 monclient: monmap: 2024-03-25T19:47:04.311 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:47:04.311 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:04.311 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:47:04.311987+0000 2024-03-25T19:47:04.311 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:47:04.311987+0000 2024-03-25T19:47:04.311 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:04.311 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:04.311 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:47:04.311 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:47:04.311 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:47:04.311 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:04.312 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.311+0000 7ff51adc7640 1 Processor -- start 2024-03-25T19:47:04.312 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.311+0000 7ff51adc7640 1 -- start start 2024-03-25T19:47:04.312 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.312+0000 7ff51adc7640 10 monclient: init 2024-03-25T19:47:04.312 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.312+0000 7ff51adc7640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:04.312 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.312+0000 7ff51adc7640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:04.313 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.312+0000 7ff51adc7640 10 monclient(hunting): picked mon.noname-c con 0x7ff5140a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:47:04.313 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.312+0000 7ff51adc7640 10 monclient(hunting): picked mon.noname-a con 0x7ff51413f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:47:04.313 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.312+0000 7ff51adc7640 10 monclient(hunting): picked mon.noname-b con 0x7ff5141430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:47:04.313 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.312+0000 7ff51adc7640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff514142520 con 0x7ff51413f710 2024-03-25T19:47:04.313 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.312+0000 7ff51adc7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5140b28a0 con 0x7ff5140a98f0 2024-03-25T19:47:04.313 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.312+0000 7ff51adc7640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5140b2a30 con 0x7ff5141430c0 2024-03-25T19:47:04.313 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.312+0000 7ff51adc7640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:04.313 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.312+0000 7ff51adc7640 10 monclient(hunting): authenticate will time out at 4259.394820s 2024-03-25T19:47:04.326 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.326+0000 7ff518b63640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff5140a98f0 0x7ff5140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:52012/0 (socket says 172.21.15.81:52012) 2024-03-25T19:47:04.326 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.326+0000 7ff518b63640 1 -- 172.21.15.81:0/2772521237 learned_addr learned my addr 172.21.15.81:0/2772521237 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:47:04.345 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.345+0000 7ff5137fe640 1 -- 172.21.15.81:0/2772521237 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403822318 0 0) 0x7ff5140b28a0 con 0x7ff5140a98f0 2024-03-25T19:47:04.345 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.345+0000 7ff5137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:04.345 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.345+0000 7ff5137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:04.345 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.345+0000 7ff5137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:04.345 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.345+0000 7ff5137fe640 10 monclient(hunting): my global_id is 4456 2024-03-25T19:47:04.345 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.345+0000 7ff5137fe640 1 -- 172.21.15.81:0/2772521237 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff504003520 con 0x7ff5140a98f0 2024-03-25T19:47:04.354 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.355+0000 7ff5137fe640 1 -- 172.21.15.81:0/2772521237 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322186868 0 0) 0x7ff5140b2a30 con 0x7ff5141430c0 2024-03-25T19:47:04.355 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.355+0000 7ff5137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:04.355 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.355+0000 7ff5137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:04.355 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.355+0000 7ff5137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:04.355 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.355+0000 7ff5137fe640 10 monclient(hunting): my global_id is 4436 2024-03-25T19:47:04.355 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.355+0000 7ff5137fe640 1 -- 172.21.15.81:0/2772521237 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5140b28a0 con 0x7ff5141430c0 2024-03-25T19:47:04.358 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.358+0000 7ff5137fe640 1 -- 172.21.15.81:0/2772521237 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644824220 0 0) 0x7ff514142520 con 0x7ff51413f710 2024-03-25T19:47:04.358 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.358+0000 7ff5137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:04.358 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.358+0000 7ff5137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:04.358 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.358+0000 7ff5137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:04.358 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.358+0000 7ff5137fe640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:47:04.358 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.358+0000 7ff5137fe640 1 -- 172.21.15.81:0/2772521237 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5140b2a30 con 0x7ff51413f710 2024-03-25T19:47:04.373 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.373+0000 7ff5137fe640 1 -- 172.21.15.81:0/2772521237 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2961877263 0 0) 0x7ff504003520 con 0x7ff5140a98f0 2024-03-25T19:47:04.373 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.373+0000 7ff5137fe640 1 -- 172.21.15.81:0/2772521237 >> v1:172.21.15.81:6790/0 conn(0x7ff5141430c0 legacy=0x7ff51414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:04.374 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.373+0000 7ff5137fe640 1 -- 172.21.15.81:0/2772521237 >> v1:172.21.15.81:6789/0 conn(0x7ff51413f710 legacy=0x7ff5141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:04.374 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.373+0000 7ff5137fe640 10 monclient: _finish_hunting 0 2024-03-25T19:47:04.374 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.373+0000 7ff5137fe640 1 monclient: found mon.noname-c 2024-03-25T19:47:04.374 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.373+0000 7ff5137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:04.374 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.373+0000 7ff5137fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:47:04.374 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.373+0000 7ff5137fe640 1 -- 172.21.15.81:0/2772521237 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5140b44c0 con 0x7ff5140a98f0 2024-03-25T19:47:04.374 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.373+0000 7ff5137fe640 10 monclient: _finish_auth 0 2024-03-25T19:47:04.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.373+0000 7ff5137fe640 10 monclient: _check_auth_tickets 2024-03-25T19:47:04.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.373+0000 7ff5137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:04.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.373+0000 7ff5137fe640 1 -- 172.21.15.81:0/2772521237 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7ff508001690 con 0x7ff5140a98f0 2024-03-25T19:47:04.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.373+0000 7ff5137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:04.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.373+0000 7ff5137fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:04.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.373+0000 7ff5137fe640 10 monclient: dump: 2024-03-25T19:47:04.375 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:04.375 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:04.375 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:04.375 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:04.375 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:04.375 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:04.375 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:04.375 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:04.375 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:04.376 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:04.376 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.373+0000 7ff51adc7640 5 monclient: authenticate success, global_id 4456 2024-03-25T19:47:04.376 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.373+0000 7ff51adc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:04.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.380+0000 7ff5137fe640 1 -- 172.21.15.81:0/2772521237 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5080017f0 con 0x7ff5140a98f0 2024-03-25T19:47:04.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.380+0000 7ff5137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:04.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.380+0000 7ff5137fe640 1 -- 172.21.15.81:0/2772521237 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7ff508001c00 con 0x7ff5140a98f0 2024-03-25T19:47:04.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.380+0000 7ff51adc7640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:04.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.380+0000 7ff51adc7640 4 set_mon_vals no callback set 2024-03-25T19:47:04.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.380+0000 7ff5137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:04.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.380+0000 7ff5137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:04.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.380+0000 7ff5137fe640 10 monclient: dump: 2024-03-25T19:47:04.381 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:04.381 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:04.381 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:04.381 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:04.381 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:04.382 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:04.382 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:04.382 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:04.382 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:04.382 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:04.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.381+0000 7ff51adc7640 10 monclient: shutdown 2024-03-25T19:47:04.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.381+0000 7ff51adc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:04.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.381+0000 7ff51adc7640 1 -- 172.21.15.81:0/2772521237 >> v1:172.21.15.98:6789/0 conn(0x7ff5140a98f0 legacy=0x7ff5140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:04.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.381+0000 7ff51adc7640 1 -- 172.21.15.81:0/2772521237 shutdown_connections 2024-03-25T19:47:04.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.381+0000 7ff51adc7640 1 -- 172.21.15.81:0/2772521237 >> 172.21.15.81:0/2772521237 conn(0x7ff5140b7ea0 msgr2=0x7ff5140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:04.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.381+0000 7ff51adc7640 1 -- 172.21.15.81:0/2772521237 shutdown_connections 2024-03-25T19:47:04.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.381+0000 7ff51adc7640 1 -- 172.21.15.81:0/2772521237 wait complete. 2024-03-25T19:47:04.384 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.382+0000 7ff51adc7640 10 monclient: build_initial_monmap 2024-03-25T19:47:04.384 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.382+0000 7ff51adc7640 10 monclient: monmap: 2024-03-25T19:47:04.384 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:47:04.384 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:04.384 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:47:04.384 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:47:04.384 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:04.384 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:47:04.385 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:47:04.385 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:47:04.385 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:47:04.385 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:04.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.382+0000 7ff51adc7640 1 librados: starting msgr at 2024-03-25T19:47:04.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.382+0000 7ff51adc7640 1 librados: starting objecter 2024-03-25T19:47:04.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.382+0000 7ff51adc7640 1 Processor -- start 2024-03-25T19:47:04.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.382+0000 7ff51adc7640 1 -- start start 2024-03-25T19:47:04.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.382+0000 7ff51adc7640 1 librados: setting wanted keys 2024-03-25T19:47:04.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.382+0000 7ff51adc7640 1 librados: calling monclient init 2024-03-25T19:47:04.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.382+0000 7ff51adc7640 10 monclient: init 2024-03-25T19:47:04.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.383+0000 7ff51adc7640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:04.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.383+0000 7ff51adc7640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:04.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.383+0000 7ff51adc7640 10 monclient(hunting): picked mon.noname-c con 0x7ff5140a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:47:04.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.383+0000 7ff51adc7640 10 monclient(hunting): picked mon.noname-b con 0x7ff51413f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:47:04.386 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.383+0000 7ff51adc7640 10 monclient(hunting): picked mon.noname-a con 0x7ff5141430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:47:04.386 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.383+0000 7ff51adc7640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5141d7380 con 0x7ff5141430c0 2024-03-25T19:47:04.386 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.383+0000 7ff51adc7640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff514139840 con 0x7ff51413f710 2024-03-25T19:47:04.386 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.383+0000 7ff51adc7640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5141d8200 con 0x7ff5140a98f0 2024-03-25T19:47:04.386 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.383+0000 7ff51adc7640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:04.386 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.383+0000 7ff51adc7640 10 monclient(hunting): authenticate will time out at 4259.465656s 2024-03-25T19:47:04.386 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.384+0000 7ff518b63640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7ff5140a98f0 0x7ff5141d7af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:55392/0 (socket says 172.21.15.81:55392) 2024-03-25T19:47:04.386 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.384+0000 7ff518b63640 1 -- 172.21.15.81:0/4289561160 learned_addr learned my addr 172.21.15.81:0/4289561160 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:47:04.386 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.384+0000 7ff5117fa640 10 client.?.objecter ms_handle_connect 0x7ff5140a98f0 2024-03-25T19:47:04.386 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.384+0000 7ff5117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:04.386 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.384+0000 7ff5117fa640 10 client.?.objecter ms_handle_connect 0x7ff51413f710 2024-03-25T19:47:04.386 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.384+0000 7ff5117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:04.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.442+0000 7ff5117fa640 10 client.?.objecter ms_handle_connect 0x7ff5141430c0 2024-03-25T19:47:04.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.442+0000 7ff5117fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:04.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.463+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248083796 0 0) 0x7ff5141d7380 con 0x7ff5141430c0 2024-03-25T19:47:04.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.463+0000 7ff5117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:04.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.463+0000 7ff5117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:04.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.463+0000 7ff5117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:04.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.463+0000 7ff5117fa640 10 monclient(hunting): my global_id is 4461 2024-03-25T19:47:04.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.463+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4f8004eb0 con 0x7ff5141430c0 2024-03-25T19:47:04.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.465+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916779131 0 0) 0x7ff514139840 con 0x7ff51413f710 2024-03-25T19:47:04.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.466+0000 7ff5117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:04.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.466+0000 7ff5117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:04.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.466+0000 7ff5117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:04.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.466+0000 7ff5117fa640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:47:04.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.466+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5141d7380 con 0x7ff51413f710 2024-03-25T19:47:04.470 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.471+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450111571 0 0) 0x7ff5141d8200 con 0x7ff5140a98f0 2024-03-25T19:47:04.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.471+0000 7ff5117fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:04.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.471+0000 7ff5117fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:04.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.471+0000 7ff5117fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:04.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.471+0000 7ff5117fa640 10 monclient(hunting): my global_id is 4439 2024-03-25T19:47:04.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.471+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff514139840 con 0x7ff5140a98f0 2024-03-25T19:47:04.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.485+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3177940293 0 0) 0x7ff5141d7380 con 0x7ff51413f710 2024-03-25T19:47:04.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.485+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5141d8200 con 0x7ff51413f710 2024-03-25T19:47:04.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.485+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7ff500001870 con 0x7ff51413f710 2024-03-25T19:47:04.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.485+0000 7ff5117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:04.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.485+0000 7ff5117fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:04.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.485+0000 7ff5117fa640 10 monclient(hunting): dump: 2024-03-25T19:47:04.486 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:04.486 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:04.486 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:04.486 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:04.486 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:04.486 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:04.486 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:04.486 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:04.486 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:04.486 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:04.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.485+0000 7ff5117fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:04.494 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.495+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 814699018 0 0) 0x7ff514139840 con 0x7ff5140a98f0 2024-03-25T19:47:04.494 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.495+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5141d7380 con 0x7ff5140a98f0 2024-03-25T19:47:04.495 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.495+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7ff508001b00 con 0x7ff5140a98f0 2024-03-25T19:47:04.495 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.495+0000 7ff5117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:04.495 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.495+0000 7ff5117fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:04.495 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.495+0000 7ff5117fa640 10 monclient(hunting): dump: 2024-03-25T19:47:04.495 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:04.495 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:04.495 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:04.495 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:04.495 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:04.495 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:04.495 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:04.495 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:04.495 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:04.495 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:04.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2605324012 0 0) 0x7ff5141d8200 con 0x7ff51413f710 2024-03-25T19:47:04.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 >> v1:172.21.15.81:6790/0 conn(0x7ff5140a98f0 legacy=0x7ff5141d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:04.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 >> v1:172.21.15.81:6789/0 conn(0x7ff5141430c0 legacy=0x7ff5141daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:04.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff5117fa640 10 monclient: _finish_hunting 0 2024-03-25T19:47:04.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff5117fa640 1 monclient: found mon.b 2024-03-25T19:47:04.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff5117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:04.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff5117fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:04.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5141d75d0 con 0x7ff51413f710 2024-03-25T19:47:04.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff5117fa640 10 monclient: _finish_auth 0 2024-03-25T19:47:04.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff5117fa640 10 monclient: _check_auth_tickets 2024-03-25T19:47:04.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff5117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:04.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff51adc7640 5 monclient: authenticate success, global_id 4459 2024-03-25T19:47:04.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff51adc7640 10 monclient: _renew_subs 2024-03-25T19:47:04.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff51adc7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:04.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff51adc7640 1 -- 172.21.15.81:0/4289561160 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5141db6a0 con 0x7ff51413f710 2024-03-25T19:47:04.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff51adc7640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:04.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff51adc7640 10 monclient: _renew_subs 2024-03-25T19:47:04.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff51adc7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:04.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff51adc7640 1 -- 172.21.15.81:0/4289561160 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff514137910 con 0x7ff51413f710 2024-03-25T19:47:04.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.496+0000 7ff51adc7640 1 librados: init done 2024-03-25T19:47:04.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.498+0000 7ff51adc7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:04.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.498+0000 7ff51adc7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:04.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.498+0000 7ff51adc7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:04.499 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.498+0000 7ff51adc7640 1 -- 172.21.15.81:0/4289561160 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5140627b0 con 0x7ff51413f710 2024-03-25T19:47:04.791 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.791+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff500003610 con 0x7ff51413f710 2024-03-25T19:47:04.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.791+0000 7ff5117fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:04.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.791+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7ff500001d80 con 0x7ff51413f710 2024-03-25T19:47:04.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.791+0000 7ff5117fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:04.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.791+0000 7ff5117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:04.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.791+0000 7ff5117fa640 10 monclient: dump: 2024-03-25T19:47:04.792 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:04.792 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:04.792 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:04.792 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:04.792 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:04.792 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:04.792 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:04.793 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:04.793 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:04.793 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:04.793 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.791+0000 7ff512ffd640 4 set_mon_vals no callback set 2024-03-25T19:47:04.793 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.792+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7ff500019790 con 0x7ff51413f710 2024-03-25T19:47:04.793 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.793+0000 7ff518b63640 10 monclient: get_auth_request con 0x7ff4f806df80 auth_method 0 2024-03-25T19:47:04.793 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.793+0000 7ff5117fa640 10 client.4459.objecter ms_handle_connect 0x7ff4f806df80 2024-03-25T19:47:04.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.797+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7ff5000535d0 con 0x7ff51413f710 2024-03-25T19:47:04.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.797+0000 7ff5117fa640 10 client.4459.objecter ms_dispatch 0x7ff514000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:04.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.798+0000 7ff5117fa640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:04.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.798+0000 7ff5117fa640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:04.798 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.798+0000 7ff5117fa640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-25T19:47:04.798 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.798+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff50004ef10 con 0x7ff51413f710 2024-03-25T19:47:04.798 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.798+0000 7ff5117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:04.798 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:04.798+0000 7ff5117fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:05.058 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.058+0000 7ff51adc7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:05.058 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.058+0000 7ff51adc7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:05.058 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.058+0000 7ff51adc7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:05.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.058+0000 7ff51adc7640 1 -- 172.21.15.81:0/4289561160 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5141d7760 con 0x7ff51413f710 2024-03-25T19:47:05.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.091+0000 7ff5117fa640 1 -- 172.21.15.81:0/4289561160 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7ff500051010 con 0x7ff51413f710 2024-03-25T19:47:05.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.091+0000 7ff5117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:05.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.091+0000 7ff5117fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:05.091 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:47:05.092 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:05.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.094+0000 7ff51adc7640 10 librados: watch_flush enter 2024-03-25T19:47:05.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.094+0000 7ff51adc7640 10 librados: watch_flush exit 2024-03-25T19:47:05.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.094+0000 7ff51adc7640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-25T19:47:05.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.094+0000 7ff51adc7640 10 client.4459.objecter successfully canceled tick 2024-03-25T19:47:05.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.094+0000 7ff51adc7640 1 -- 172.21.15.81:0/4289561160 >> v1:172.21.15.98:6800/3980829775 conn(0x7ff4f806df80 legacy=0x7ff4f8070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:05.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.094+0000 7ff51adc7640 10 monclient: shutdown 2024-03-25T19:47:05.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.094+0000 7ff51adc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:05.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.094+0000 7ff51adc7640 1 -- 172.21.15.81:0/4289561160 >> v1:172.21.15.98:6789/0 conn(0x7ff51413f710 legacy=0x7ff5141da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:05.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.094+0000 7ff51adc7640 1 -- 172.21.15.81:0/4289561160 shutdown_connections 2024-03-25T19:47:05.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.094+0000 7ff51adc7640 1 -- 172.21.15.81:0/4289561160 >> 172.21.15.81:0/4289561160 conn(0x7ff5140b7ea0 msgr2=0x7ff514074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:05.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.094+0000 7ff51adc7640 1 -- 172.21.15.81:0/4289561160 shutdown_connections 2024-03-25T19:47:05.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.094+0000 7ff51adc7640 1 -- 172.21.15.81:0/4289561160 wait complete. 2024-03-25T19:47:05.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:05.094+0000 7ff51adc7640 1 librados: shutdown 2024-03-25T19:47:05.106 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:11.911 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:12.025 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.025+0000 7f33df220640 10 monclient: get_monmap_and_config 2024-03-25T19:47:12.025 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.025+0000 7f33df220640 10 monclient: build_initial_monmap 2024-03-25T19:47:12.025 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.025+0000 7f33df220640 10 monclient: monmap: 2024-03-25T19:47:12.025 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:47:12.025 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:12.025 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:47:12.026658+0000 2024-03-25T19:47:12.025 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:47:12.026658+0000 2024-03-25T19:47:12.026 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:12.026 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:12.026 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:47:12.026 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:47:12.026 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:47:12.026 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:12.026 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.026+0000 7f33df220640 1 Processor -- start 2024-03-25T19:47:12.026 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.026+0000 7f33df220640 1 -- start start 2024-03-25T19:47:12.026 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.026+0000 7f33df220640 10 monclient: init 2024-03-25T19:47:12.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.027+0000 7f33df220640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:12.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.027+0000 7f33df220640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:47:12.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.027+0000 7f33df220640 10 monclient(hunting): picked mon.noname-c con 0x7f33d80b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:47:12.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.027+0000 7f33df220640 10 monclient(hunting): picked mon.noname-b con 0x7f33d80b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:47:12.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.027+0000 7f33df220640 10 monclient(hunting): picked mon.noname-a con 0x7f33d814be90 addr v1:172.21.15.81:6789/0 2024-03-25T19:47:12.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.027+0000 7f33df220640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d80af940 con 0x7f33d814be90 2024-03-25T19:47:12.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.027+0000 7f33df220640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d80b0a70 con 0x7f33d80b4d60 2024-03-25T19:47:12.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.027+0000 7f33df220640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d80b0d50 con 0x7f33d80b1730 2024-03-25T19:47:12.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.027+0000 7f33df220640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:12.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.027+0000 7f33df220640 10 monclient(hunting): authenticate will time out at 4267.109628s 2024-03-25T19:47:12.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.043+0000 7f33dcfbc640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f33d80b4d60 0x7f33d813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:38612/0 (socket says 172.21.15.81:38612) 2024-03-25T19:47:12.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.043+0000 7f33dcfbc640 1 -- 172.21.15.81:0/3360824326 learned_addr learned my addr 172.21.15.81:0/3360824326 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:47:12.054 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.055+0000 7f33d77fe640 1 -- 172.21.15.81:0/3360824326 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153565763 0 0) 0x7f33d80b0a70 con 0x7f33d80b4d60 2024-03-25T19:47:12.054 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.055+0000 7f33d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:12.054 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.055+0000 7f33d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:12.054 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.055+0000 7f33d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:12.054 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.055+0000 7f33d77fe640 10 monclient(hunting): my global_id is 4462 2024-03-25T19:47:12.054 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.055+0000 7f33d77fe640 1 -- 172.21.15.81:0/3360824326 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33c8003520 con 0x7f33d80b4d60 2024-03-25T19:47:12.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.059+0000 7f33d77fe640 1 -- 172.21.15.81:0/3360824326 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284238577 0 0) 0x7f33d80af940 con 0x7f33d814be90 2024-03-25T19:47:12.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.059+0000 7f33d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:12.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.059+0000 7f33d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:12.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.059+0000 7f33d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:12.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.059+0000 7f33d77fe640 10 monclient(hunting): my global_id is 4464 2024-03-25T19:47:12.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.059+0000 7f33d77fe640 1 -- 172.21.15.81:0/3360824326 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33d80b0a70 con 0x7f33d814be90 2024-03-25T19:47:12.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.061+0000 7f33d77fe640 1 -- 172.21.15.81:0/3360824326 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543451681 0 0) 0x7f33d80b0d50 con 0x7f33d80b1730 2024-03-25T19:47:12.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.061+0000 7f33d77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:12.061 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.061+0000 7f33d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:12.061 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.061+0000 7f33d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:12.061 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.061+0000 7f33d77fe640 10 monclient(hunting): my global_id is 4442 2024-03-25T19:47:12.061 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.061+0000 7f33d77fe640 1 -- 172.21.15.81:0/3360824326 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33d80af940 con 0x7f33d80b1730 2024-03-25T19:47:12.065 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33d77fe640 1 -- 172.21.15.81:0/3360824326 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 90964308 0 0) 0x7f33c8003520 con 0x7f33d80b4d60 2024-03-25T19:47:12.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33d77fe640 1 -- 172.21.15.81:0/3360824326 >> v1:172.21.15.81:6790/0 conn(0x7f33d80b1730 legacy=0x7f33d80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:12.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33d77fe640 1 -- 172.21.15.81:0/3360824326 >> v1:172.21.15.81:6789/0 conn(0x7f33d814be90 legacy=0x7f33d814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:12.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33d77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:47:12.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33d77fe640 1 monclient: found mon.noname-c 2024-03-25T19:47:12.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:12.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33d77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:47:12.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33d77fe640 1 -- 172.21.15.81:0/3360824326 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d80b1030 con 0x7f33d80b4d60 2024-03-25T19:47:12.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33d77fe640 10 monclient: _finish_auth 0 2024-03-25T19:47:12.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33d77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:47:12.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:12.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33d77fe640 1 -- 172.21.15.81:0/3360824326 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f33cc001690 con 0x7f33d80b4d60 2024-03-25T19:47:12.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33df220640 5 monclient: authenticate success, global_id 4462 2024-03-25T19:47:12.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33df220640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:12.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:12.066 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33d77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:12.067 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33d77fe640 10 monclient: dump: 2024-03-25T19:47:12.067 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:12.067 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:12.067 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:12.067 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:12.067 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:12.067 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:12.067 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:12.067 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:12.067 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:12.067 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:12.067 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.065+0000 7f33df220640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:12.073 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.073+0000 7f33d77fe640 1 -- 172.21.15.81:0/3360824326 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33cc0017f0 con 0x7f33d80b4d60 2024-03-25T19:47:12.073 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.073+0000 7f33d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:12.073 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.073+0000 7f33d77fe640 1 -- 172.21.15.81:0/3360824326 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f33cc001c00 con 0x7f33d80b4d60 2024-03-25T19:47:12.073 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.073+0000 7f33d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:12.073 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.073+0000 7f33d77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:12.073 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.073+0000 7f33d77fe640 10 monclient: dump: 2024-03-25T19:47:12.073 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:12.073 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:12.073 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:12.073 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:12.073 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:12.073 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:12.073 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:12.074 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:12.074 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:12.074 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:12.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.073+0000 7f33df220640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:12.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.073+0000 7f33df220640 4 set_mon_vals no callback set 2024-03-25T19:47:12.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.074+0000 7f33df220640 10 monclient: shutdown 2024-03-25T19:47:12.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.074+0000 7f33df220640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:12.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.074+0000 7f33df220640 1 -- 172.21.15.81:0/3360824326 >> v1:172.21.15.98:6789/0 conn(0x7f33d80b4d60 legacy=0x7f33d813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:12.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.074+0000 7f33df220640 1 -- 172.21.15.81:0/3360824326 shutdown_connections 2024-03-25T19:47:12.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.074+0000 7f33df220640 1 -- 172.21.15.81:0/3360824326 >> 172.21.15.81:0/3360824326 conn(0x7f33d80617a0 msgr2=0x7f33d8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:12.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.074+0000 7f33df220640 1 -- 172.21.15.81:0/3360824326 shutdown_connections 2024-03-25T19:47:12.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.074+0000 7f33df220640 1 -- 172.21.15.81:0/3360824326 wait complete. 2024-03-25T19:47:12.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.075+0000 7f33df220640 10 monclient: build_initial_monmap 2024-03-25T19:47:12.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.075+0000 7f33df220640 10 monclient: monmap: 2024-03-25T19:47:12.076 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:47:12.076 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:12.076 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:47:12.076 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:47:12.076 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:12.076 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:47:12.076 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:47:12.076 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:47:12.076 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:47:12.076 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:12.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.076+0000 7f33df220640 1 librados: starting msgr at 2024-03-25T19:47:12.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.076+0000 7f33df220640 1 librados: starting objecter 2024-03-25T19:47:12.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.076+0000 7f33df220640 1 Processor -- start 2024-03-25T19:47:12.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.076+0000 7f33df220640 1 -- start start 2024-03-25T19:47:12.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.076+0000 7f33df220640 1 librados: setting wanted keys 2024-03-25T19:47:12.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.076+0000 7f33df220640 1 librados: calling monclient init 2024-03-25T19:47:12.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.076+0000 7f33df220640 10 monclient: init 2024-03-25T19:47:12.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.077+0000 7f33df220640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:12.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.077+0000 7f33df220640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:12.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.077+0000 7f33df220640 10 monclient(hunting): picked mon.noname-a con 0x7f33d80b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:47:12.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.077+0000 7f33df220640 10 monclient(hunting): picked mon.noname-b con 0x7f33d80b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:47:12.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.077+0000 7f33df220640 10 monclient(hunting): picked mon.noname-c con 0x7f33d814be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:47:12.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.077+0000 7f33df220640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d8141e50 con 0x7f33d80b1730 2024-03-25T19:47:12.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.077+0000 7f33df220640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d81dad80 con 0x7f33d80b4d60 2024-03-25T19:47:12.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.077+0000 7f33df220640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33d81420a0 con 0x7f33d814be90 2024-03-25T19:47:12.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.077+0000 7f33df220640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:12.079 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.077+0000 7f33df220640 10 monclient(hunting): authenticate will time out at 4267.159321s 2024-03-25T19:47:12.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.084+0000 7f33dcfbc640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f33d80b1730 0x7f33d8138690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:45530/0 (socket says 172.21.15.81:45530) 2024-03-25T19:47:12.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.084+0000 7f33dcfbc640 1 -- 172.21.15.81:0/1429610141 learned_addr learned my addr 172.21.15.81:0/1429610141 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:47:12.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.085+0000 7f33d57fa640 10 client.?.objecter ms_handle_connect 0x7f33d80b1730 2024-03-25T19:47:12.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.085+0000 7f33d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:12.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.091+0000 7f33d57fa640 10 client.?.objecter ms_handle_connect 0x7f33d80b4d60 2024-03-25T19:47:12.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.091+0000 7f33d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:12.096 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.097+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719071512 0 0) 0x7f33d8141e50 con 0x7f33d80b1730 2024-03-25T19:47:12.096 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.097+0000 7f33d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:12.096 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.097+0000 7f33d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:12.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.097+0000 7f33d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:12.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.097+0000 7f33d57fa640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:47:12.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.097+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33bc004eb0 con 0x7f33d80b1730 2024-03-25T19:47:12.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.098+0000 7f33d57fa640 10 client.?.objecter ms_handle_connect 0x7f33d814be90 2024-03-25T19:47:12.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.098+0000 7f33d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:12.103 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.104+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662156868 0 0) 0x7f33d81dad80 con 0x7f33d80b4d60 2024-03-25T19:47:12.103 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.104+0000 7f33d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:12.103 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.104+0000 7f33d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:12.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.104+0000 7f33d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:12.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.104+0000 7f33d57fa640 10 monclient(hunting): my global_id is 4465 2024-03-25T19:47:12.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.104+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33d8141e50 con 0x7f33d80b4d60 2024-03-25T19:47:12.108 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.109+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1815419579 0 0) 0x7f33d81420a0 con 0x7f33d814be90 2024-03-25T19:47:12.109 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.109+0000 7f33d57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:12.109 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.109+0000 7f33d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:12.109 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.109+0000 7f33d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:12.109 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.109+0000 7f33d57fa640 10 monclient(hunting): my global_id is 4445 2024-03-25T19:47:12.109 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.109+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33d81dad80 con 0x7f33d814be90 2024-03-25T19:47:12.116 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.116+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3602319430 0 0) 0x7f33bc004eb0 con 0x7f33d80b1730 2024-03-25T19:47:12.116 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.116+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33d81420a0 con 0x7f33d80b1730 2024-03-25T19:47:12.116 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.116+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f33cc001bc0 con 0x7f33d80b1730 2024-03-25T19:47:12.116 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.116+0000 7f33d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:12.116 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.116+0000 7f33d57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:12.116 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.116+0000 7f33d57fa640 10 monclient(hunting): dump: 2024-03-25T19:47:12.116 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:12.116 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:12.116 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:12.116 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:12.116 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:12.117 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:12.117 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:12.117 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:12.117 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:12.117 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:12.117 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.116+0000 7f33d57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:12.125 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.125+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 532982401 0 0) 0x7f33d8141e50 con 0x7f33d80b4d60 2024-03-25T19:47:12.125 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.125+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33bc004eb0 con 0x7f33d80b4d60 2024-03-25T19:47:12.125 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.125+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f33c4001870 con 0x7f33d80b4d60 2024-03-25T19:47:12.125 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.125+0000 7f33d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:12.125 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.125+0000 7f33d57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:12.125 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.125+0000 7f33d57fa640 10 monclient(hunting): dump: 2024-03-25T19:47:12.126 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:12.126 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:12.126 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:12.126 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:12.126 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:12.126 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:12.126 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:12.126 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:12.126 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:12.126 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:12.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2437853964 0 0) 0x7f33d81420a0 con 0x7f33d80b1730 2024-03-25T19:47:12.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 >> v1:172.21.15.81:6790/0 conn(0x7f33d814be90 legacy=0x7f33d8142c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:12.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 >> v1:172.21.15.98:6789/0 conn(0x7f33d80b4d60 legacy=0x7f33d81db210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:12.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33d57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:47:12.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33d57fa640 1 monclient: found mon.a 2024-03-25T19:47:12.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33d57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:12.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33d57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:47:12.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d81db920 con 0x7f33d80b1730 2024-03-25T19:47:12.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33d57fa640 10 monclient: _finish_auth 0 2024-03-25T19:47:12.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33d57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:47:12.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:12.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33df220640 5 monclient: authenticate success, global_id 4467 2024-03-25T19:47:12.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33df220640 10 monclient: _renew_subs 2024-03-25T19:47:12.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33df220640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:47:12.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33df220640 1 -- 172.21.15.81:0/1429610141 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d81dbb70 con 0x7f33d80b1730 2024-03-25T19:47:12.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33df220640 10 client.4467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:12.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33df220640 10 monclient: _renew_subs 2024-03-25T19:47:12.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33df220640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:47:12.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33df220640 1 -- 172.21.15.81:0/1429610141 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d81427b0 con 0x7f33d80b1730 2024-03-25T19:47:12.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.127+0000 7f33df220640 1 librados: init done 2024-03-25T19:47:12.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.128+0000 7f33df220640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:12.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.128+0000 7f33df220640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:12.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.128+0000 7f33df220640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:47:12.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.128+0000 7f33df220640 1 -- 172.21.15.81:0/1429610141 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d80627b0 con 0x7f33d80b1730 2024-03-25T19:47:12.445 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.445+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33cc004a90 con 0x7f33d80b1730 2024-03-25T19:47:12.445 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.445+0000 7f33d57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:12.445 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.445+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f33cc004da0 con 0x7f33d80b1730 2024-03-25T19:47:12.445 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.445+0000 7f33d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:12.445 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.445+0000 7f33d57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:12.445 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.445+0000 7f33d57fa640 10 monclient: dump: 2024-03-25T19:47:12.446 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:12.446 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:12.446 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:12.446 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:12.446 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:12.446 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:12.446 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:12.446 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:12.446 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:12.446 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:12.446 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.445+0000 7f33d6ffd640 4 set_mon_vals no callback set 2024-03-25T19:47:12.447 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.446+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f33cc01ad70 con 0x7f33d80b1730 2024-03-25T19:47:12.447 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.447+0000 7f33d7fff640 10 monclient: get_auth_request con 0x7f33bc06df70 auth_method 0 2024-03-25T19:47:12.447 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.447+0000 7f33d57fa640 10 client.4467.objecter ms_handle_connect 0x7f33bc06df70 2024-03-25T19:47:12.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.454+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f33cc054930 con 0x7f33d80b1730 2024-03-25T19:47:12.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.454+0000 7f33d57fa640 10 client.4467.objecter ms_dispatch 0x7f33d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:12.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.454+0000 7f33d57fa640 3 client.4467.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:12.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.454+0000 7f33d57fa640 3 client.4467.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:12.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.454+0000 7f33d57fa640 20 client.4467.objecter dump_active .. 0 homeless 2024-03-25T19:47:12.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.454+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33cc050440 con 0x7f33d80b1730 2024-03-25T19:47:12.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.454+0000 7f33d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:12.455 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.454+0000 7f33d57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:12.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.696+0000 7f33df220640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:12.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.696+0000 7f33df220640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:12.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.696+0000 7f33df220640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:47:12.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.696+0000 7f33df220640 1 -- 172.21.15.81:0/1429610141 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33d8145010 con 0x7f33d80b1730 2024-03-25T19:47:12.716 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.717+0000 7f33d57fa640 1 -- 172.21.15.81:0/1429610141 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f33cc052370 con 0x7f33d80b1730 2024-03-25T19:47:12.716 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.717+0000 7f33d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:12.717 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.717+0000 7f33d57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:12.717 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:47:12.717 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:12.719 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.719+0000 7f33df220640 10 librados: watch_flush enter 2024-03-25T19:47:12.720 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.719+0000 7f33df220640 10 librados: watch_flush exit 2024-03-25T19:47:12.720 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.719+0000 7f33df220640 20 client.4467.objecter shutdown clearing up homeless session... 2024-03-25T19:47:12.720 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.719+0000 7f33df220640 10 client.4467.objecter successfully canceled tick 2024-03-25T19:47:12.720 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.719+0000 7f33df220640 1 -- 172.21.15.81:0/1429610141 >> v1:172.21.15.98:6800/3980829775 conn(0x7f33bc06df70 legacy=0x7f33bc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:12.720 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.719+0000 7f33df220640 10 monclient: shutdown 2024-03-25T19:47:12.720 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.719+0000 7f33df220640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:12.720 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.719+0000 7f33df220640 1 -- 172.21.15.81:0/1429610141 >> v1:172.21.15.81:6789/0 conn(0x7f33d80b1730 legacy=0x7f33d8138690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:12.720 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.720+0000 7f33df220640 1 -- 172.21.15.81:0/1429610141 shutdown_connections 2024-03-25T19:47:12.720 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.720+0000 7f33df220640 1 -- 172.21.15.81:0/1429610141 >> 172.21.15.81:0/1429610141 conn(0x7f33d80617a0 msgr2=0x7f33d813c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:12.720 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.720+0000 7f33df220640 1 -- 172.21.15.81:0/1429610141 shutdown_connections 2024-03-25T19:47:12.720 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.720+0000 7f33df220640 1 -- 172.21.15.81:0/1429610141 wait complete. 2024-03-25T19:47:12.720 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:12.720+0000 7f33df220640 1 librados: shutdown 2024-03-25T19:47:12.731 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:17.518 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:05:40.160 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:17.566 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:05:40.207 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:17.609 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:05:40.251 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:17.621 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:05:40.262 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:17.631 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:05:40.272 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:17.645 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:40.279 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:17.650 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:40.285 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:17.659 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:05:40.300 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:17.669 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:05:40.304 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:19.538 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:19.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.642+0000 7efc45f4d640 10 monclient: get_monmap_and_config 2024-03-25T19:47:19.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.642+0000 7efc45f4d640 10 monclient: build_initial_monmap 2024-03-25T19:47:19.642 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.642+0000 7efc45f4d640 10 monclient: monmap: 2024-03-25T19:47:19.642 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:47:19.642 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:19.642 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:47:19.642964+0000 2024-03-25T19:47:19.642 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:47:19.642964+0000 2024-03-25T19:47:19.642 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:19.642 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:19.642 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:47:19.642 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:47:19.642 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:47:19.642 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:19.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.642+0000 7efc45f4d640 1 Processor -- start 2024-03-25T19:47:19.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.642+0000 7efc45f4d640 1 -- start start 2024-03-25T19:47:19.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.642+0000 7efc45f4d640 10 monclient: init 2024-03-25T19:47:19.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.643+0000 7efc45f4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:19.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.643+0000 7efc45f4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:19.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.643+0000 7efc45f4d640 10 monclient(hunting): picked mon.noname-b con 0x7efc400b10d0 addr v1:172.21.15.81:6790/0 2024-03-25T19:47:19.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.643+0000 7efc45f4d640 10 monclient(hunting): picked mon.noname-a con 0x7efc400b4360 addr v1:172.21.15.81:6789/0 2024-03-25T19:47:19.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.643+0000 7efc45f4d640 10 monclient(hunting): picked mon.noname-c con 0x7efc400a9470 addr v1:172.21.15.98:6789/0 2024-03-25T19:47:19.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.643+0000 7efc45f4d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc400a9ab0 con 0x7efc400b4360 2024-03-25T19:47:19.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.643+0000 7efc45f4d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc400b2400 con 0x7efc400a9470 2024-03-25T19:47:19.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.643+0000 7efc45f4d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc400aa2a0 con 0x7efc400b10d0 2024-03-25T19:47:19.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.643+0000 7efc45f4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:19.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.643+0000 7efc45f4d640 10 monclient(hunting): authenticate will time out at 4274.725395s 2024-03-25T19:47:19.651 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.651+0000 7efc3ffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7efc400a9470 legacy=0x7efc4014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-25T19:47:19.651 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.651+0000 7efc3ffff640 1 -- >> v1:172.21.15.98:6789/0 conn(0x7efc400a9470 legacy=0x7efc4014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T19:47:19.652 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.651+0000 7efc3ffff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7efc400a9470 0x7efc4014dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2024-03-25T19:47:19.653 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.653+0000 7efc3f7fe640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7efc400b10d0 0x7efc400b4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:49078/0 (socket says 172.21.15.81:49078) 2024-03-25T19:47:19.653 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.653+0000 7efc3f7fe640 1 -- 172.21.15.81:0/746908520 learned_addr learned my addr 172.21.15.81:0/746908520 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:47:19.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.673+0000 7efc3e7fc640 1 -- 172.21.15.81:0/746908520 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744210532 0 0) 0x7efc400aa2a0 con 0x7efc400b10d0 2024-03-25T19:47:19.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.673+0000 7efc3e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:19.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.673+0000 7efc3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:19.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.673+0000 7efc3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:19.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.673+0000 7efc3e7fc640 10 monclient(hunting): my global_id is 4448 2024-03-25T19:47:19.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.673+0000 7efc3e7fc640 1 -- 172.21.15.81:0/746908520 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc28003520 con 0x7efc400b10d0 2024-03-25T19:47:19.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.741+0000 7efc3e7fc640 1 -- 172.21.15.81:0/746908520 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625621464 0 0) 0x7efc400a9ab0 con 0x7efc400b4360 2024-03-25T19:47:19.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.741+0000 7efc3e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:19.742 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.741+0000 7efc3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:19.742 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.741+0000 7efc3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:19.742 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.741+0000 7efc3e7fc640 10 monclient(hunting): my global_id is 4470 2024-03-25T19:47:19.742 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.741+0000 7efc3e7fc640 1 -- 172.21.15.81:0/746908520 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc400aa2a0 con 0x7efc400b4360 2024-03-25T19:47:19.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.760+0000 7efc3e7fc640 1 -- 172.21.15.81:0/746908520 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2845633918 0 0) 0x7efc400aa2a0 con 0x7efc400b4360 2024-03-25T19:47:19.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.760+0000 7efc3e7fc640 1 -- 172.21.15.81:0/746908520 >> v1:172.21.15.81:6790/0 conn(0x7efc400b10d0 legacy=0x7efc400b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:19.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.760+0000 7efc3e7fc640 1 -- 172.21.15.81:0/746908520 >> v1:172.21.15.98:6789/0 conn(0x7efc400a9470 legacy=0x7efc4014dbc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:47:19.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.760+0000 7efc3e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:47:19.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.760+0000 7efc3e7fc640 1 monclient: found mon.noname-a 2024-03-25T19:47:19.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.760+0000 7efc3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:19.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.761+0000 7efc3e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:47:19.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.761+0000 7efc3e7fc640 1 -- 172.21.15.81:0/746908520 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc400b14b0 con 0x7efc400b4360 2024-03-25T19:47:19.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.761+0000 7efc3e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:47:19.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.761+0000 7efc3e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:47:19.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.761+0000 7efc3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:19.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.761+0000 7efc3e7fc640 1 -- 172.21.15.81:0/746908520 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7efc34001670 con 0x7efc400b4360 2024-03-25T19:47:19.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.761+0000 7efc3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:19.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.761+0000 7efc3e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:19.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.761+0000 7efc3e7fc640 10 monclient: dump: 2024-03-25T19:47:19.762 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:19.762 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:19.762 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:19.762 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:19.762 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:19.762 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:19.762 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:19.762 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:19.762 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:19.762 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:19.763 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.761+0000 7efc45f4d640 5 monclient: authenticate success, global_id 4470 2024-03-25T19:47:19.763 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.761+0000 7efc45f4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:19.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.792+0000 7efc3e7fc640 1 -- 172.21.15.81:0/746908520 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc340017d0 con 0x7efc400b4360 2024-03-25T19:47:19.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.792+0000 7efc3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:19.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.792+0000 7efc3e7fc640 1 -- 172.21.15.81:0/746908520 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7efc34001c20 con 0x7efc400b4360 2024-03-25T19:47:19.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.792+0000 7efc45f4d640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:19.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.792+0000 7efc45f4d640 4 set_mon_vals no callback set 2024-03-25T19:47:19.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.792+0000 7efc3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:19.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.792+0000 7efc3e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:19.792 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.792+0000 7efc3e7fc640 10 monclient: dump: 2024-03-25T19:47:19.792 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:19.792 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:19.793 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:19.793 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:19.793 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:19.793 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:19.793 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:19.793 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:19.793 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:19.793 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:19.795 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.792+0000 7efc45f4d640 10 monclient: shutdown 2024-03-25T19:47:19.795 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.792+0000 7efc45f4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:19.795 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.792+0000 7efc45f4d640 1 -- 172.21.15.81:0/746908520 >> v1:172.21.15.81:6789/0 conn(0x7efc400b4360 legacy=0x7efc400b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:19.795 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.793+0000 7efc45f4d640 1 -- 172.21.15.81:0/746908520 shutdown_connections 2024-03-25T19:47:19.795 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.793+0000 7efc45f4d640 1 -- 172.21.15.81:0/746908520 >> 172.21.15.81:0/746908520 conn(0x7efc400617a0 msgr2=0x7efc40061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:19.795 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.793+0000 7efc45f4d640 1 -- 172.21.15.81:0/746908520 shutdown_connections 2024-03-25T19:47:19.795 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.793+0000 7efc45f4d640 1 -- 172.21.15.81:0/746908520 wait complete. 2024-03-25T19:47:19.795 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.793+0000 7efc45f4d640 10 monclient: build_initial_monmap 2024-03-25T19:47:19.795 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.793+0000 7efc45f4d640 10 monclient: monmap: 2024-03-25T19:47:19.796 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:47:19.796 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:19.796 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:47:19.796 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:47:19.796 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:19.796 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:47:19.796 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:47:19.796 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:47:19.796 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:47:19.796 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:19.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.794+0000 7efc45f4d640 1 librados: starting msgr at 2024-03-25T19:47:19.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.794+0000 7efc45f4d640 1 librados: starting objecter 2024-03-25T19:47:19.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.794+0000 7efc45f4d640 1 Processor -- start 2024-03-25T19:47:19.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.794+0000 7efc45f4d640 1 -- start start 2024-03-25T19:47:19.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.794+0000 7efc45f4d640 1 librados: setting wanted keys 2024-03-25T19:47:19.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.794+0000 7efc45f4d640 1 librados: calling monclient init 2024-03-25T19:47:19.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.794+0000 7efc45f4d640 10 monclient: init 2024-03-25T19:47:19.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.794+0000 7efc45f4d640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:19.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.794+0000 7efc45f4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:19.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.794+0000 7efc45f4d640 10 monclient(hunting): picked mon.noname-a con 0x7efc400a9470 addr v1:172.21.15.81:6789/0 2024-03-25T19:47:19.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.794+0000 7efc45f4d640 10 monclient(hunting): picked mon.noname-c con 0x7efc400b10d0 addr v1:172.21.15.81:6790/0 2024-03-25T19:47:19.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.794+0000 7efc45f4d640 10 monclient(hunting): picked mon.noname-b con 0x7efc400b4360 addr v1:172.21.15.98:6789/0 2024-03-25T19:47:19.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.795+0000 7efc45f4d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc40134980 con 0x7efc400a9470 2024-03-25T19:47:19.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.795+0000 7efc45f4d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc401426c0 con 0x7efc400b4360 2024-03-25T19:47:19.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.795+0000 7efc45f4d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4013b030 con 0x7efc400b10d0 2024-03-25T19:47:19.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.795+0000 7efc45f4d640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:19.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.795+0000 7efc45f4d640 10 monclient(hunting): authenticate will time out at 4274.876936s 2024-03-25T19:47:19.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.805+0000 7efc3f7fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7efc400a9470 0x7efc40141fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:34338/0 (socket says 172.21.15.81:34338) 2024-03-25T19:47:19.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.805+0000 7efc3f7fe640 1 -- 172.21.15.81:0/2419760549 learned_addr learned my addr 172.21.15.81:0/2419760549 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:47:19.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.808+0000 7efc27fff640 10 client.?.objecter ms_handle_connect 0x7efc400a9470 2024-03-25T19:47:19.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.808+0000 7efc27fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:19.813 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.813+0000 7efc27fff640 10 client.?.objecter ms_handle_connect 0x7efc400b4360 2024-03-25T19:47:19.813 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.813+0000 7efc27fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:19.817 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.817+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280046394 0 0) 0x7efc401426c0 con 0x7efc400b4360 2024-03-25T19:47:19.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.817+0000 7efc27fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:19.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.817+0000 7efc27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:19.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.817+0000 7efc27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:19.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.817+0000 7efc27fff640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:47:19.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.817+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc2c004eb0 con 0x7efc400b4360 2024-03-25T19:47:19.825 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.826+0000 7efc27fff640 10 client.?.objecter ms_handle_connect 0x7efc400b10d0 2024-03-25T19:47:19.825 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.826+0000 7efc27fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:19.829 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.829+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130823426 0 0) 0x7efc40134980 con 0x7efc400a9470 2024-03-25T19:47:19.829 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.829+0000 7efc27fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:19.829 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.829+0000 7efc27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:19.829 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.829+0000 7efc27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:19.829 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.829+0000 7efc27fff640 10 monclient(hunting): my global_id is 4473 2024-03-25T19:47:19.829 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.829+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc401426c0 con 0x7efc400a9470 2024-03-25T19:47:19.835 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.835+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3076972068 0 0) 0x7efc2c004eb0 con 0x7efc400b4360 2024-03-25T19:47:19.835 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.835+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc40134980 con 0x7efc400b4360 2024-03-25T19:47:19.835 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.835+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7efc38001b40 con 0x7efc400b4360 2024-03-25T19:47:19.835 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.835+0000 7efc27fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:19.835 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.835+0000 7efc27fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:19.835 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.835+0000 7efc27fff640 10 monclient(hunting): dump: 2024-03-25T19:47:19.835 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:19.835 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:19.835 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:19.835 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:19.835 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:19.835 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:19.835 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:19.835 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:19.835 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:19.836 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:19.836 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.835+0000 7efc27fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:19.837 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.838+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897220805 0 0) 0x7efc4013b030 con 0x7efc400b10d0 2024-03-25T19:47:19.838 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.838+0000 7efc27fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:19.838 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.838+0000 7efc27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:19.838 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.838+0000 7efc27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:19.838 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.838+0000 7efc27fff640 10 monclient(hunting): my global_id is 4451 2024-03-25T19:47:19.838 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.838+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc2c004eb0 con 0x7efc400b10d0 2024-03-25T19:47:19.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 352869291 0 0) 0x7efc40134980 con 0x7efc400b4360 2024-03-25T19:47:19.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 >> v1:172.21.15.81:6790/0 conn(0x7efc400b10d0 legacy=0x7efc40134e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:19.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 >> v1:172.21.15.81:6789/0 conn(0x7efc400a9470 legacy=0x7efc40141fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:19.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc27fff640 10 monclient: _finish_hunting 0 2024-03-25T19:47:19.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc27fff640 1 monclient: found mon.b 2024-03-25T19:47:19.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:19.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc27fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:19.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc401412a0 con 0x7efc400b4360 2024-03-25T19:47:19.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc27fff640 10 monclient: _finish_auth 0 2024-03-25T19:47:19.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc27fff640 10 monclient: _check_auth_tickets 2024-03-25T19:47:19.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:19.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc45f4d640 5 monclient: authenticate success, global_id 4468 2024-03-25T19:47:19.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc45f4d640 10 monclient: _renew_subs 2024-03-25T19:47:19.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc45f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:19.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc45f4d640 1 -- 172.21.15.81:0/2419760549 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc40141550 con 0x7efc400b4360 2024-03-25T19:47:19.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc45f4d640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:19.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc45f4d640 10 monclient: _renew_subs 2024-03-25T19:47:19.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc45f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:19.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc45f4d640 1 -- 172.21.15.81:0/2419760549 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc401e2ee0 con 0x7efc400b4360 2024-03-25T19:47:19.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.851+0000 7efc45f4d640 1 librados: init done 2024-03-25T19:47:19.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.852+0000 7efc45f4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:19.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.852+0000 7efc45f4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:19.853 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.852+0000 7efc45f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:19.853 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:19.852+0000 7efc45f4d640 1 -- 172.21.15.81:0/2419760549 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc400627b0 con 0x7efc400b4360 2024-03-25T19:47:20.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.132+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc38002050 con 0x7efc400b4360 2024-03-25T19:47:20.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.132+0000 7efc27fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:20.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.132+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7efc38004cb0 con 0x7efc400b4360 2024-03-25T19:47:20.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.132+0000 7efc27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:20.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.132+0000 7efc27fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:20.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.132+0000 7efc27fff640 10 monclient: dump: 2024-03-25T19:47:20.132 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:20.132 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:20.133 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:20.133 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:20.133 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:20.133 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:20.133 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:20.133 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:20.133 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:20.133 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:20.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.132+0000 7efc3dffb640 4 set_mon_vals no callback set 2024-03-25T19:47:20.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.133+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7efc3801ac80 con 0x7efc400b4360 2024-03-25T19:47:20.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.134+0000 7efc3f7fe640 10 monclient: get_auth_request con 0x7efc2c06ddd0 auth_method 0 2024-03-25T19:47:20.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.134+0000 7efc27fff640 10 client.4468.objecter ms_handle_connect 0x7efc2c06ddd0 2024-03-25T19:47:20.139 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.139+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7efc38054960 con 0x7efc400b4360 2024-03-25T19:47:20.139 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.139+0000 7efc27fff640 10 client.4468.objecter ms_dispatch 0x7efc40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:20.139 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.139+0000 7efc27fff640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:20.139 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.139+0000 7efc27fff640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:20.139 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.139+0000 7efc27fff640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-25T19:47:20.140 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.140+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc380502a0 con 0x7efc400b4360 2024-03-25T19:47:20.140 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.140+0000 7efc27fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:20.140 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.140+0000 7efc27fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:20.386 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.386+0000 7efc45f4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:20.387 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.386+0000 7efc45f4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:20.387 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.386+0000 7efc45f4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:20.387 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.386+0000 7efc45f4d640 1 -- 172.21.15.81:0/2419760549 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc400b6cc0 con 0x7efc400b4360 2024-03-25T19:47:20.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.420+0000 7efc27fff640 1 -- 172.21.15.81:0/2419760549 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7efc380523a0 con 0x7efc400b4360 2024-03-25T19:47:20.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.420+0000 7efc27fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:20.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.420+0000 7efc27fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:20.420 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:47:20.420 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:20.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.422+0000 7efc45f4d640 10 librados: watch_flush enter 2024-03-25T19:47:20.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.422+0000 7efc45f4d640 10 librados: watch_flush exit 2024-03-25T19:47:20.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.422+0000 7efc45f4d640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-25T19:47:20.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.422+0000 7efc45f4d640 10 client.4468.objecter successfully canceled tick 2024-03-25T19:47:20.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.422+0000 7efc45f4d640 1 -- 172.21.15.81:0/2419760549 >> v1:172.21.15.98:6800/3980829775 conn(0x7efc2c06ddd0 legacy=0x7efc2c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:20.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.422+0000 7efc45f4d640 10 monclient: shutdown 2024-03-25T19:47:20.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.422+0000 7efc45f4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:20.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.422+0000 7efc45f4d640 1 -- 172.21.15.81:0/2419760549 >> v1:172.21.15.98:6789/0 conn(0x7efc400b4360 legacy=0x7efc40135520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:20.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.422+0000 7efc45f4d640 1 -- 172.21.15.81:0/2419760549 shutdown_connections 2024-03-25T19:47:20.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.422+0000 7efc45f4d640 1 -- 172.21.15.81:0/2419760549 >> 172.21.15.81:0/2419760549 conn(0x7efc400617a0 msgr2=0x7efc40061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:20.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.422+0000 7efc45f4d640 1 -- 172.21.15.81:0/2419760549 shutdown_connections 2024-03-25T19:47:20.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.422+0000 7efc45f4d640 1 -- 172.21.15.81:0/2419760549 wait complete. 2024-03-25T19:47:20.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:20.423+0000 7efc45f4d640 1 librados: shutdown 2024-03-25T19:47:20.436 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:27.242 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:27.356 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.357+0000 7f36357d6640 10 monclient: get_monmap_and_config 2024-03-25T19:47:27.357 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.357+0000 7f36357d6640 10 monclient: build_initial_monmap 2024-03-25T19:47:27.357 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.357+0000 7f36357d6640 10 monclient: monmap: 2024-03-25T19:47:27.357 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:47:27.357 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:27.357 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:47:27.358089+0000 2024-03-25T19:47:27.357 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:47:27.358089+0000 2024-03-25T19:47:27.357 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:27.357 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:27.357 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:47:27.357 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:47:27.357 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:47:27.358 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:27.359 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.357+0000 7f36357d6640 1 Processor -- start 2024-03-25T19:47:27.359 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.357+0000 7f36357d6640 1 -- start start 2024-03-25T19:47:27.359 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.357+0000 7f36357d6640 10 monclient: init 2024-03-25T19:47:27.359 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.358+0000 7f36357d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:27.359 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.358+0000 7f36357d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:27.359 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.358+0000 7f36357d6640 10 monclient(hunting): picked mon.noname-a con 0x7f36300a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:47:27.359 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.358+0000 7f36357d6640 10 monclient(hunting): picked mon.noname-b con 0x7f363013f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:47:27.359 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.358+0000 7f36357d6640 10 monclient(hunting): picked mon.noname-c con 0x7f36301430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:47:27.359 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.358+0000 7f36357d6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3630142520 con 0x7f36300a98f0 2024-03-25T19:47:27.359 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.358+0000 7f36357d6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36300b28a0 con 0x7f36301430c0 2024-03-25T19:47:27.360 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.358+0000 7f36357d6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36300b2a30 con 0x7f363013f710 2024-03-25T19:47:27.360 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.358+0000 7f36357d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:27.360 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.358+0000 7f36357d6640 10 monclient(hunting): authenticate will time out at 4282.440392s 2024-03-25T19:47:27.364 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.364+0000 7f362f7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f36301430c0 0x7f363014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:33062/0 (socket says 172.21.15.81:33062) 2024-03-25T19:47:27.364 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.364+0000 7f362f7fe640 1 -- 172.21.15.81:0/2669670244 learned_addr learned my addr 172.21.15.81:0/2669670244 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:47:27.452 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.453+0000 7f362e7fc640 1 -- 172.21.15.81:0/2669670244 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011237853 0 0) 0x7f3630142520 con 0x7f36300a98f0 2024-03-25T19:47:27.452 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.453+0000 7f362e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:27.453 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.453+0000 7f362e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:27.453 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.453+0000 7f362e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:27.453 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.453+0000 7f362e7fc640 10 monclient(hunting): my global_id is 4476 2024-03-25T19:47:27.453 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.453+0000 7f362e7fc640 1 -- 172.21.15.81:0/2669670244 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f361c003520 con 0x7f36300a98f0 2024-03-25T19:47:27.462 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.462+0000 7f362e7fc640 1 -- 172.21.15.81:0/2669670244 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3725952984 0 0) 0x7f361c003520 con 0x7f36300a98f0 2024-03-25T19:47:27.462 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.462+0000 7f362e7fc640 1 -- 172.21.15.81:0/2669670244 >> v1:172.21.15.81:6790/0 conn(0x7f363013f710 legacy=0x7f36301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:27.462 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.462+0000 7f362e7fc640 1 -- 172.21.15.81:0/2669670244 >> v1:172.21.15.98:6789/0 conn(0x7f36301430c0 legacy=0x7f363014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:27.462 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.462+0000 7f362e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:47:27.462 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.462+0000 7f362e7fc640 1 monclient: found mon.noname-a 2024-03-25T19:47:27.462 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.462+0000 7f362e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:27.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.462+0000 7f362e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:47:27.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.462+0000 7f362e7fc640 1 -- 172.21.15.81:0/2669670244 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36300b44c0 con 0x7f36300a98f0 2024-03-25T19:47:27.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.462+0000 7f362e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:47:27.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.462+0000 7f362e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:47:27.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.462+0000 7f362e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:27.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.462+0000 7f362e7fc640 1 -- 172.21.15.81:0/2669670244 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3620001690 con 0x7f36300a98f0 2024-03-25T19:47:27.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.462+0000 7f36357d6640 5 monclient: authenticate success, global_id 4476 2024-03-25T19:47:27.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.462+0000 7f362e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:27.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.462+0000 7f362e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:27.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.462+0000 7f362e7fc640 10 monclient: dump: 2024-03-25T19:47:27.463 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:27.463 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:27.463 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:27.463 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:27.463 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:27.464 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:27.464 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:27.464 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:27.464 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:27.464 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:27.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.462+0000 7f36357d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:27.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.489+0000 7f362e7fc640 1 -- 172.21.15.81:0/2669670244 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36200017f0 con 0x7f36300a98f0 2024-03-25T19:47:27.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.489+0000 7f362e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:27.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.489+0000 7f362e7fc640 1 -- 172.21.15.81:0/2669670244 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3620001c00 con 0x7f36300a98f0 2024-03-25T19:47:27.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.489+0000 7f362e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:27.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.489+0000 7f362e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:27.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.489+0000 7f362e7fc640 10 monclient: dump: 2024-03-25T19:47:27.489 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:27.489 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:27.489 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:27.489 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:27.489 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:27.489 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:27.489 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:27.489 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:27.490 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:27.490 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:27.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.489+0000 7f36357d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:27.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.489+0000 7f36357d6640 4 set_mon_vals no callback set 2024-03-25T19:47:27.491 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.490+0000 7f36357d6640 10 monclient: shutdown 2024-03-25T19:47:27.491 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.490+0000 7f36357d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:27.491 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.490+0000 7f36357d6640 1 -- 172.21.15.81:0/2669670244 >> v1:172.21.15.81:6789/0 conn(0x7f36300a98f0 legacy=0x7f36300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:27.491 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.490+0000 7f36357d6640 1 -- 172.21.15.81:0/2669670244 shutdown_connections 2024-03-25T19:47:27.491 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.490+0000 7f36357d6640 1 -- 172.21.15.81:0/2669670244 >> 172.21.15.81:0/2669670244 conn(0x7f36300b7ea0 msgr2=0x7f36300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:27.492 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.490+0000 7f36357d6640 1 -- 172.21.15.81:0/2669670244 shutdown_connections 2024-03-25T19:47:27.492 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.490+0000 7f36357d6640 1 -- 172.21.15.81:0/2669670244 wait complete. 2024-03-25T19:47:27.492 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.491+0000 7f36357d6640 10 monclient: build_initial_monmap 2024-03-25T19:47:27.492 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.491+0000 7f36357d6640 10 monclient: monmap: 2024-03-25T19:47:27.492 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:47:27.492 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:27.492 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:47:27.492 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:47:27.492 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:27.492 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:47:27.492 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:47:27.492 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:47:27.492 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:47:27.492 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:27.492 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.491+0000 7f36357d6640 1 librados: starting msgr at 2024-03-25T19:47:27.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.491+0000 7f36357d6640 1 librados: starting objecter 2024-03-25T19:47:27.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.491+0000 7f36357d6640 1 Processor -- start 2024-03-25T19:47:27.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.492+0000 7f36357d6640 1 -- start start 2024-03-25T19:47:27.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.492+0000 7f36357d6640 1 librados: setting wanted keys 2024-03-25T19:47:27.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.492+0000 7f36357d6640 1 librados: calling monclient init 2024-03-25T19:47:27.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.492+0000 7f36357d6640 10 monclient: init 2024-03-25T19:47:27.494 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.492+0000 7f36357d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:27.494 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.492+0000 7f36357d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:27.494 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.492+0000 7f36357d6640 10 monclient(hunting): picked mon.noname-c con 0x7f36300a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:47:27.494 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.493+0000 7f36357d6640 10 monclient(hunting): picked mon.noname-b con 0x7f363013f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:47:27.494 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.493+0000 7f36357d6640 10 monclient(hunting): picked mon.noname-a con 0x7f36301430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:47:27.494 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.493+0000 7f36357d6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3630135ac0 con 0x7f36301430c0 2024-03-25T19:47:27.494 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.493+0000 7f36357d6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3630137030 con 0x7f363013f710 2024-03-25T19:47:27.495 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.493+0000 7f36357d6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36301372a0 con 0x7f36300a98f0 2024-03-25T19:47:27.495 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.493+0000 7f36357d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:27.495 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.493+0000 7f36357d6640 10 monclient(hunting): authenticate will time out at 4282.574906s 2024-03-25T19:47:27.499 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.500+0000 7f36267fc640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f363013f710 0x7f36301379a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:33118/0 (socket says 172.21.15.81:33118) 2024-03-25T19:47:27.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.500+0000 7f36267fc640 1 -- 172.21.15.81:0/3944738746 learned_addr learned my addr 172.21.15.81:0/3944738746 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:47:27.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.501+0000 7f3627fff640 10 client.?.objecter ms_handle_connect 0x7f363013f710 2024-03-25T19:47:27.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.501+0000 7f3627fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:27.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.524+0000 7f3627fff640 10 client.?.objecter ms_handle_connect 0x7f36300a98f0 2024-03-25T19:47:27.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.524+0000 7f3627fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:27.530 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.531+0000 7f3627fff640 10 client.?.objecter ms_handle_connect 0x7f36301430c0 2024-03-25T19:47:27.531 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.531+0000 7f3627fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:27.531 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.532+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404283979 0 0) 0x7f36301372a0 con 0x7f36300a98f0 2024-03-25T19:47:27.531 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.532+0000 7f3627fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:27.531 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.532+0000 7f3627fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:27.532 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.532+0000 7f3627fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:27.532 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.532+0000 7f3627fff640 10 monclient(hunting): my global_id is 4457 2024-03-25T19:47:27.532 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.532+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3610004eb0 con 0x7f36300a98f0 2024-03-25T19:47:27.544 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.545+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477297962 0 0) 0x7f3630137030 con 0x7f363013f710 2024-03-25T19:47:27.544 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.545+0000 7f3627fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:27.544 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.545+0000 7f3627fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:27.544 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.545+0000 7f3627fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:27.544 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.545+0000 7f3627fff640 10 monclient(hunting): my global_id is 4471 2024-03-25T19:47:27.545 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.545+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36301372a0 con 0x7f363013f710 2024-03-25T19:47:27.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.557+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4264336445 0 0) 0x7f3610004eb0 con 0x7f36300a98f0 2024-03-25T19:47:27.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.557+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3630137030 con 0x7f36300a98f0 2024-03-25T19:47:27.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.557+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3620001c00 con 0x7f36300a98f0 2024-03-25T19:47:27.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.557+0000 7f3627fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:27.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.557+0000 7f3627fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:27.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.557+0000 7f3627fff640 10 monclient(hunting): dump: 2024-03-25T19:47:27.558 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:27.558 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:27.558 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:27.558 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:27.558 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:27.558 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:27.558 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:27.558 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:27.558 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:27.558 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:27.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.557+0000 7f3627fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:27.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.558+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269955426 0 0) 0x7f3630135ac0 con 0x7f36301430c0 2024-03-25T19:47:27.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.558+0000 7f3627fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:27.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.558+0000 7f3627fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:27.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.558+0000 7f3627fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:27.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.558+0000 7f3627fff640 10 monclient(hunting): my global_id is 4479 2024-03-25T19:47:27.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.558+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3610004eb0 con 0x7f36301430c0 2024-03-25T19:47:27.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.566+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3347778817 0 0) 0x7f36301372a0 con 0x7f363013f710 2024-03-25T19:47:27.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.566+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3630135ac0 con 0x7f363013f710 2024-03-25T19:47:27.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.566+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3618001970 con 0x7f363013f710 2024-03-25T19:47:27.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.566+0000 7f3627fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:27.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.566+0000 7f3627fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:27.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.566+0000 7f3627fff640 10 monclient(hunting): dump: 2024-03-25T19:47:27.566 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:27.566 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:27.566 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:27.566 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:27.567 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:27.567 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:27.567 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:27.567 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:27.567 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:27.567 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:27.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.585+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 904765046 0 0) 0x7f3610004eb0 con 0x7f36301430c0 2024-03-25T19:47:27.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.585+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36301372a0 con 0x7f36301430c0 2024-03-25T19:47:27.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.585+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3628001a60 con 0x7f36301430c0 2024-03-25T19:47:27.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.585+0000 7f3627fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:27.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.585+0000 7f3627fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:27.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.585+0000 7f3627fff640 10 monclient(hunting): dump: 2024-03-25T19:47:27.586 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:27.586 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:27.586 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:27.586 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:27.586 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:27.586 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:27.586 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:27.586 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:27.586 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:27.586 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:27.588 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4236178098 0 0) 0x7f3630135ac0 con 0x7f363013f710 2024-03-25T19:47:27.588 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 >> v1:172.21.15.81:6790/0 conn(0x7f36300a98f0 legacy=0x7f3630133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:27.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 >> v1:172.21.15.81:6789/0 conn(0x7f36301430c0 legacy=0x7f36301e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:27.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f3627fff640 10 monclient: _finish_hunting 0 2024-03-25T19:47:27.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f3627fff640 1 monclient: found mon.b 2024-03-25T19:47:27.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f3627fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:27.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f3627fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:27.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3630137580 con 0x7f363013f710 2024-03-25T19:47:27.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f3627fff640 10 monclient: _finish_auth 0 2024-03-25T19:47:27.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f3627fff640 10 monclient: _check_auth_tickets 2024-03-25T19:47:27.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f3627fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:27.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f36357d6640 5 monclient: authenticate success, global_id 4471 2024-03-25T19:47:27.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f36357d6640 10 monclient: _renew_subs 2024-03-25T19:47:27.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f36357d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:27.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f36357d6640 1 -- 172.21.15.81:0/3944738746 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36301363b0 con 0x7f363013f710 2024-03-25T19:47:27.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f36357d6640 10 client.4471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:27.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f36357d6640 10 monclient: _renew_subs 2024-03-25T19:47:27.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f36357d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:27.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f36357d6640 1 -- 172.21.15.81:0/3944738746 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36301353a0 con 0x7f363013f710 2024-03-25T19:47:27.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.588+0000 7f36357d6640 1 librados: init done 2024-03-25T19:47:27.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.589+0000 7f36357d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:27.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.589+0000 7f36357d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:27.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.589+0000 7f36357d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:27.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.589+0000 7f36357d6640 1 -- 172.21.15.81:0/3944738746 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3630130d90 con 0x7f363013f710 2024-03-25T19:47:27.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.856+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3618003610 con 0x7f363013f710 2024-03-25T19:47:27.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.856+0000 7f3627fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:27.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.856+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3618003800 con 0x7f363013f710 2024-03-25T19:47:27.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.856+0000 7f3627fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:27.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.856+0000 7f3627fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:27.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.856+0000 7f3627fff640 10 monclient: dump: 2024-03-25T19:47:27.856 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:27.856 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:27.856 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:27.856 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:27.856 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:27.856 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:27.857 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:27.857 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:27.857 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:27.857 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:27.857 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.856+0000 7f362dffb640 4 set_mon_vals no callback set 2024-03-25T19:47:27.857 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.857+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f3618003a20 con 0x7f363013f710 2024-03-25T19:47:27.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.857+0000 7f362effd640 10 monclient: get_auth_request con 0x7f361006e0f0 auth_method 0 2024-03-25T19:47:27.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.858+0000 7f3627fff640 10 client.4471.objecter ms_handle_connect 0x7f361006e0f0 2024-03-25T19:47:27.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.861+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f36180545e0 con 0x7f363013f710 2024-03-25T19:47:27.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.862+0000 7f3627fff640 10 client.4471.objecter ms_dispatch 0x7f3630000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:27.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.862+0000 7f3627fff640 3 client.4471.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:27.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.862+0000 7f3627fff640 3 client.4471.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:27.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.862+0000 7f3627fff640 20 client.4471.objecter dump_active .. 0 homeless 2024-03-25T19:47:27.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.862+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36180500f0 con 0x7f363013f710 2024-03-25T19:47:27.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.862+0000 7f3627fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:27.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:27.862+0000 7f3627fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:28.106 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.106+0000 7f36357d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:28.106 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.106+0000 7f36357d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:28.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.107+0000 7f36357d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:28.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.107+0000 7f36357d6640 1 -- 172.21.15.81:0/3944738746 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3630074cc0 con 0x7f363013f710 2024-03-25T19:47:28.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.133+0000 7f3627fff640 1 -- 172.21.15.81:0/3944738746 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f3618052020 con 0x7f363013f710 2024-03-25T19:47:28.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.133+0000 7f3627fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:28.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.133+0000 7f3627fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:28.133 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:47:28.133 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:28.136 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.135+0000 7f36357d6640 10 librados: watch_flush enter 2024-03-25T19:47:28.136 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.136+0000 7f36357d6640 10 librados: watch_flush exit 2024-03-25T19:47:28.136 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.136+0000 7f36357d6640 20 client.4471.objecter shutdown clearing up homeless session... 2024-03-25T19:47:28.136 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.136+0000 7f36357d6640 10 client.4471.objecter successfully canceled tick 2024-03-25T19:47:28.136 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.136+0000 7f36357d6640 1 -- 172.21.15.81:0/3944738746 >> v1:172.21.15.98:6800/3980829775 conn(0x7f361006e0f0 legacy=0x7f36100705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:28.136 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.136+0000 7f36357d6640 10 monclient: shutdown 2024-03-25T19:47:28.136 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.136+0000 7f36357d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:28.136 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.136+0000 7f36357d6640 1 -- 172.21.15.81:0/3944738746 >> v1:172.21.15.98:6789/0 conn(0x7f363013f710 legacy=0x7f36301379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:28.137 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.136+0000 7f36357d6640 1 -- 172.21.15.81:0/3944738746 shutdown_connections 2024-03-25T19:47:28.137 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.136+0000 7f36357d6640 1 -- 172.21.15.81:0/3944738746 >> 172.21.15.81:0/3944738746 conn(0x7f36300b7ea0 msgr2=0x7f3630074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:28.137 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.136+0000 7f36357d6640 1 -- 172.21.15.81:0/3944738746 shutdown_connections 2024-03-25T19:47:28.137 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.136+0000 7f36357d6640 1 -- 172.21.15.81:0/3944738746 wait complete. 2024-03-25T19:47:28.137 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:28.136+0000 7f36357d6640 1 librados: shutdown 2024-03-25T19:47:28.148 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:34.955 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:35.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.070+0000 7f46798ba640 10 monclient: get_monmap_and_config 2024-03-25T19:47:35.070 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.070+0000 7f46798ba640 10 monclient: build_initial_monmap 2024-03-25T19:47:35.070 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.070+0000 7f46798ba640 10 monclient: monmap: 2024-03-25T19:47:35.070 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:47:35.070 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:35.070 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:47:35.071169+0000 2024-03-25T19:47:35.070 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:47:35.071169+0000 2024-03-25T19:47:35.070 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:35.070 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:35.070 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:47:35.070 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:47:35.071 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:47:35.071 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:35.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.071+0000 7f46798ba640 1 Processor -- start 2024-03-25T19:47:35.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.071+0000 7f46798ba640 1 -- start start 2024-03-25T19:47:35.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.071+0000 7f46798ba640 10 monclient: init 2024-03-25T19:47:35.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.071+0000 7f46798ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:35.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.071+0000 7f46798ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:35.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.071+0000 7f46798ba640 10 monclient(hunting): picked mon.noname-b con 0x7f46740a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:47:35.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.071+0000 7f46798ba640 10 monclient(hunting): picked mon.noname-c con 0x7f467413f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:47:35.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.071+0000 7f46798ba640 10 monclient(hunting): picked mon.noname-a con 0x7f46741430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:47:35.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.071+0000 7f46798ba640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4674142520 con 0x7f46741430c0 2024-03-25T19:47:35.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.071+0000 7f46798ba640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46740b28a0 con 0x7f467413f710 2024-03-25T19:47:35.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.071+0000 7f46798ba640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46740b2a30 con 0x7f46740a98f0 2024-03-25T19:47:35.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.071+0000 7f46798ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:35.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.071+0000 7f46798ba640 10 monclient(hunting): authenticate will time out at 4290.153615s 2024-03-25T19:47:35.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.075+0000 7f466bfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f467413f710 0x7f46741429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:51846/0 (socket says 172.21.15.81:51846) 2024-03-25T19:47:35.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.075+0000 7f466bfff640 1 -- 172.21.15.81:0/3958687471 learned_addr learned my addr 172.21.15.81:0/3958687471 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:47:35.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.097+0000 7f46727fc640 1 -- 172.21.15.81:0/3958687471 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523232829 0 0) 0x7f46740b28a0 con 0x7f467413f710 2024-03-25T19:47:35.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.098+0000 7f46727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:35.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.098+0000 7f46727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:35.098 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.098+0000 7f46727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:35.098 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.098+0000 7f46727fc640 10 monclient(hunting): my global_id is 4474 2024-03-25T19:47:35.098 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.098+0000 7f46727fc640 1 -- 172.21.15.81:0/3958687471 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4660003520 con 0x7f467413f710 2024-03-25T19:47:35.099 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.099+0000 7f46727fc640 1 -- 172.21.15.81:0/3958687471 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544244513 0 0) 0x7f46740b2a30 con 0x7f46740a98f0 2024-03-25T19:47:35.099 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.099+0000 7f46727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:35.099 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.099+0000 7f46727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:35.099 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.099+0000 7f46727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:35.099 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.099+0000 7f46727fc640 10 monclient(hunting): my global_id is 4460 2024-03-25T19:47:35.099 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.099+0000 7f46727fc640 1 -- 172.21.15.81:0/3958687471 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46740b28a0 con 0x7f46740a98f0 2024-03-25T19:47:35.101 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.102+0000 7f46727fc640 1 -- 172.21.15.81:0/3958687471 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157965562 0 0) 0x7f4674142520 con 0x7f46741430c0 2024-03-25T19:47:35.101 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.102+0000 7f46727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:35.101 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.102+0000 7f46727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:35.102 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.102+0000 7f46727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:35.102 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.102+0000 7f46727fc640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:47:35.102 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.102+0000 7f46727fc640 1 -- 172.21.15.81:0/3958687471 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46740b2a30 con 0x7f46741430c0 2024-03-25T19:47:35.118 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.118+0000 7f46727fc640 1 -- 172.21.15.81:0/3958687471 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1086302727 0 0) 0x7f46740b2a30 con 0x7f46741430c0 2024-03-25T19:47:35.118 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.118+0000 7f46727fc640 1 -- 172.21.15.81:0/3958687471 >> v1:172.21.15.81:6790/0 conn(0x7f46740a98f0 legacy=0x7f46740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:35.118 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.118+0000 7f46727fc640 1 -- 172.21.15.81:0/3958687471 >> v1:172.21.15.98:6789/0 conn(0x7f467413f710 legacy=0x7f46741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:35.118 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.118+0000 7f46727fc640 10 monclient: _finish_hunting 0 2024-03-25T19:47:35.118 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.118+0000 7f46727fc640 1 monclient: found mon.noname-a 2024-03-25T19:47:35.118 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.118+0000 7f46727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:35.118 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.118+0000 7f46727fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:47:35.118 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.118+0000 7f46727fc640 1 -- 172.21.15.81:0/3958687471 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46740b44c0 con 0x7f46741430c0 2024-03-25T19:47:35.118 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.118+0000 7f46727fc640 10 monclient: _finish_auth 0 2024-03-25T19:47:35.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.118+0000 7f46727fc640 10 monclient: _check_auth_tickets 2024-03-25T19:47:35.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.118+0000 7f46727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:35.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.118+0000 7f46727fc640 1 -- 172.21.15.81:0/3958687471 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f466c001930 con 0x7f46741430c0 2024-03-25T19:47:35.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.118+0000 7f46727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:35.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.118+0000 7f46727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:35.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.118+0000 7f46727fc640 10 monclient: dump: 2024-03-25T19:47:35.119 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:35.119 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:35.119 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:35.119 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:35.119 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:35.119 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:35.119 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:35.119 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:35.119 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:35.119 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:35.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.118+0000 7f46798ba640 5 monclient: authenticate success, global_id 4482 2024-03-25T19:47:35.120 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.118+0000 7f46798ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:35.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.128+0000 7f46727fc640 1 -- 172.21.15.81:0/3958687471 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f466c001b10 con 0x7f46741430c0 2024-03-25T19:47:35.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.128+0000 7f46727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:35.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.128+0000 7f46727fc640 1 -- 172.21.15.81:0/3958687471 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f466c001f60 con 0x7f46741430c0 2024-03-25T19:47:35.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.128+0000 7f46798ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:35.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.128+0000 7f46798ba640 4 set_mon_vals no callback set 2024-03-25T19:47:35.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.128+0000 7f46727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:35.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.128+0000 7f46727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:35.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.128+0000 7f46727fc640 10 monclient: dump: 2024-03-25T19:47:35.129 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:35.129 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:35.129 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:35.129 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:35.129 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:35.129 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:35.129 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:35.129 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:35.129 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:35.129 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:35.130 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.129+0000 7f46798ba640 10 monclient: shutdown 2024-03-25T19:47:35.130 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.129+0000 7f46798ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:35.131 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.129+0000 7f46798ba640 1 -- 172.21.15.81:0/3958687471 >> v1:172.21.15.81:6789/0 conn(0x7f46741430c0 legacy=0x7f467414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:35.131 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.129+0000 7f46798ba640 1 -- 172.21.15.81:0/3958687471 shutdown_connections 2024-03-25T19:47:35.131 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.129+0000 7f46798ba640 1 -- 172.21.15.81:0/3958687471 >> 172.21.15.81:0/3958687471 conn(0x7f46740b7ea0 msgr2=0x7f46740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:35.131 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.129+0000 7f46798ba640 1 -- 172.21.15.81:0/3958687471 shutdown_connections 2024-03-25T19:47:35.131 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.129+0000 7f46798ba640 1 -- 172.21.15.81:0/3958687471 wait complete. 2024-03-25T19:47:35.131 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.130+0000 7f46798ba640 10 monclient: build_initial_monmap 2024-03-25T19:47:35.131 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.130+0000 7f46798ba640 10 monclient: monmap: 2024-03-25T19:47:35.131 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:47:35.131 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:35.131 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:47:35.131 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:47:35.131 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:35.131 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:47:35.131 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:47:35.131 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:47:35.131 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:47:35.132 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:35.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.130+0000 7f46798ba640 1 librados: starting msgr at 2024-03-25T19:47:35.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.130+0000 7f46798ba640 1 librados: starting objecter 2024-03-25T19:47:35.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.130+0000 7f46798ba640 1 Processor -- start 2024-03-25T19:47:35.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.131+0000 7f46798ba640 1 -- start start 2024-03-25T19:47:35.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.131+0000 7f46798ba640 1 librados: setting wanted keys 2024-03-25T19:47:35.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.131+0000 7f46798ba640 1 librados: calling monclient init 2024-03-25T19:47:35.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.131+0000 7f46798ba640 10 monclient: init 2024-03-25T19:47:35.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.131+0000 7f46798ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:35.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.131+0000 7f46798ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:35.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.131+0000 7f46798ba640 10 monclient(hunting): picked mon.noname-a con 0x7f46740a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:47:35.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.131+0000 7f46798ba640 10 monclient(hunting): picked mon.noname-c con 0x7f467413f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:47:35.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.131+0000 7f46798ba640 10 monclient(hunting): picked mon.noname-b con 0x7f46741430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:47:35.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.131+0000 7f46798ba640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46741339d0 con 0x7f46740a98f0 2024-03-25T19:47:35.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.131+0000 7f46798ba640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4674139970 con 0x7f46741430c0 2024-03-25T19:47:35.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.131+0000 7f46798ba640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4674137060 con 0x7f467413f710 2024-03-25T19:47:35.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.131+0000 7f46798ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:35.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.131+0000 7f46798ba640 10 monclient(hunting): authenticate will time out at 4290.213798s 2024-03-25T19:47:35.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.134+0000 7f4672ffd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f46740a98f0 0x7f4674136950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:50688/0 (socket says 172.21.15.81:50688) 2024-03-25T19:47:35.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.134+0000 7f4672ffd640 1 -- 172.21.15.81:0/4090335943 learned_addr learned my addr 172.21.15.81:0/4090335943 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:47:35.135 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.135+0000 7f4670ff9640 10 client.?.objecter ms_handle_connect 0x7f46740a98f0 2024-03-25T19:47:35.135 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.135+0000 7f4670ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:35.141 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.142+0000 7f4670ff9640 1 -- 172.21.15.81:0/4090335943 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940170059 0 0) 0x7f46741339d0 con 0x7f46740a98f0 2024-03-25T19:47:35.141 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.142+0000 7f4670ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:35.141 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.142+0000 7f4670ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:35.141 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.142+0000 7f4670ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:35.142 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.142+0000 7f4670ff9640 10 monclient(hunting): my global_id is 4485 2024-03-25T19:47:35.142 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.142+0000 7f4670ff9640 1 -- 172.21.15.81:0/4090335943 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4654004eb0 con 0x7f46740a98f0 2024-03-25T19:47:35.142 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.143+0000 7f4670ff9640 10 client.?.objecter ms_handle_connect 0x7f46741430c0 2024-03-25T19:47:35.142 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.143+0000 7f4670ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:35.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.155+0000 7f4670ff9640 1 -- 172.21.15.81:0/4090335943 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3148627203 0 0) 0x7f4654004eb0 con 0x7f46740a98f0 2024-03-25T19:47:35.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.155+0000 7f4670ff9640 1 -- 172.21.15.81:0/4090335943 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46741339d0 con 0x7f46740a98f0 2024-03-25T19:47:35.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.155+0000 7f4670ff9640 1 -- 172.21.15.81:0/4090335943 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4664001990 con 0x7f46740a98f0 2024-03-25T19:47:35.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.155+0000 7f4670ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:35.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.155+0000 7f4670ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:35.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.155+0000 7f4670ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:35.155 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:35.155 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:35.156 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:35.156 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:35.156 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:35.156 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:35.156 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:35.156 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:35.156 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:35.156 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:35.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.155+0000 7f4670ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:35.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.160+0000 7f4670ff9640 1 -- 172.21.15.81:0/4090335943 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036090274 0 0) 0x7f4674139970 con 0x7f46741430c0 2024-03-25T19:47:35.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.160+0000 7f4670ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:35.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.160+0000 7f4670ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:35.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.160+0000 7f4670ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:35.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.160+0000 7f4670ff9640 10 monclient(hunting): my global_id is 4477 2024-03-25T19:47:35.160 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.160+0000 7f4670ff9640 1 -- 172.21.15.81:0/4090335943 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4654004eb0 con 0x7f46741430c0 2024-03-25T19:47:35.160 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.161+0000 7f4670ff9640 10 client.?.objecter ms_handle_connect 0x7f467413f710 2024-03-25T19:47:35.160 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.161+0000 7f4670ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:35.161 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.161+0000 7f4670ff9640 1 -- 172.21.15.81:0/4090335943 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2633534811 0 0) 0x7f46741339d0 con 0x7f46740a98f0 2024-03-25T19:47:35.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.161+0000 7f4670ff9640 1 -- 172.21.15.81:0/4090335943 >> v1:172.21.15.81:6790/0 conn(0x7f467413f710 legacy=0x7f4674132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:35.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.161+0000 7f4670ff9640 1 -- 172.21.15.81:0/4090335943 >> v1:172.21.15.98:6789/0 conn(0x7f46741430c0 legacy=0x7f46741332c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:35.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.161+0000 7f4670ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:47:35.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.161+0000 7f4670ff9640 1 monclient: found mon.a 2024-03-25T19:47:35.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.161+0000 7f4670ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:35.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.161+0000 7f4670ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:47:35.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.161+0000 7f4670ff9640 1 -- 172.21.15.81:0/4090335943 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467413b360 con 0x7f46740a98f0 2024-03-25T19:47:35.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.161+0000 7f4670ff9640 10 monclient: _finish_auth 0 2024-03-25T19:47:35.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.161+0000 7f4670ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:47:35.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.161+0000 7f4670ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:35.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.161+0000 7f46798ba640 5 monclient: authenticate success, global_id 4485 2024-03-25T19:47:35.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.161+0000 7f46798ba640 10 monclient: _renew_subs 2024-03-25T19:47:35.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.162+0000 7f46798ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:47:35.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.162+0000 7f46798ba640 1 -- 172.21.15.81:0/4090335943 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4674137440 con 0x7f46740a98f0 2024-03-25T19:47:35.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.162+0000 7f46798ba640 10 client.4485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:35.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.162+0000 7f46798ba640 10 monclient: _renew_subs 2024-03-25T19:47:35.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.162+0000 7f46798ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:47:35.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.162+0000 7f46798ba640 1 -- 172.21.15.81:0/4090335943 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f467414e7c0 con 0x7f46740a98f0 2024-03-25T19:47:35.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.162+0000 7f46798ba640 1 librados: init done 2024-03-25T19:47:35.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.162+0000 7f46798ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:35.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.162+0000 7f46798ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:35.164 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.162+0000 7f46798ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:47:35.164 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.162+0000 7f46798ba640 1 -- 172.21.15.81:0/4090335943 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46740b3ef0 con 0x7f46740a98f0 2024-03-25T19:47:35.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.474+0000 7f4670ff9640 1 -- 172.21.15.81:0/4090335943 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4664001c70 con 0x7f46740a98f0 2024-03-25T19:47:35.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.474+0000 7f4670ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:35.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.474+0000 7f4670ff9640 1 -- 172.21.15.81:0/4090335943 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4664003a60 con 0x7f46740a98f0 2024-03-25T19:47:35.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.474+0000 7f4670ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:35.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.474+0000 7f4670ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:35.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.474+0000 7f4670ff9640 10 monclient: dump: 2024-03-25T19:47:35.474 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:35.474 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:35.474 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:35.474 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:35.474 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:35.474 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:35.475 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:35.475 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:35.475 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:35.475 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:35.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.474+0000 7f4671ffb640 4 set_mon_vals no callback set 2024-03-25T19:47:35.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.475+0000 7f4670ff9640 1 -- 172.21.15.81:0/4090335943 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f4664003c80 con 0x7f46740a98f0 2024-03-25T19:47:35.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.475+0000 7f466bfff640 10 monclient: get_auth_request con 0x7f465406dd90 auth_method 0 2024-03-25T19:47:35.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.476+0000 7f4670ff9640 10 client.4485.objecter ms_handle_connect 0x7f465406dd90 2024-03-25T19:47:35.483 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.484+0000 7f4670ff9640 1 -- 172.21.15.81:0/4090335943 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f4664054810 con 0x7f46740a98f0 2024-03-25T19:47:35.483 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.484+0000 7f4670ff9640 10 client.4485.objecter ms_dispatch 0x7f4674000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:35.483 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.484+0000 7f4670ff9640 3 client.4485.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:35.483 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.484+0000 7f4670ff9640 3 client.4485.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:35.484 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.484+0000 7f4670ff9640 20 client.4485.objecter dump_active .. 0 homeless 2024-03-25T19:47:35.484 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.484+0000 7f4670ff9640 1 -- 172.21.15.81:0/4090335943 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4664050320 con 0x7f46740a98f0 2024-03-25T19:47:35.484 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.484+0000 7f4670ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:35.484 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.484+0000 7f4670ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:35.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.729+0000 7f46798ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:35.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.729+0000 7f46798ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:35.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.729+0000 7f46798ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:47:35.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.729+0000 7f46798ba640 1 -- 172.21.15.81:0/4090335943 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f467413baf0 con 0x7f46740a98f0 2024-03-25T19:47:35.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.750+0000 7f4670ff9640 1 -- 172.21.15.81:0/4090335943 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f4664052250 con 0x7f46740a98f0 2024-03-25T19:47:35.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.750+0000 7f4670ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:35.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.750+0000 7f4670ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:35.750 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:47:35.750 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:35.752 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.752+0000 7f46798ba640 10 librados: watch_flush enter 2024-03-25T19:47:35.752 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.752+0000 7f46798ba640 10 librados: watch_flush exit 2024-03-25T19:47:35.752 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.752+0000 7f46798ba640 20 client.4485.objecter shutdown clearing up homeless session... 2024-03-25T19:47:35.752 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.752+0000 7f46798ba640 10 client.4485.objecter successfully canceled tick 2024-03-25T19:47:35.752 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.752+0000 7f46798ba640 1 -- 172.21.15.81:0/4090335943 >> v1:172.21.15.98:6800/3980829775 conn(0x7f465406dd90 legacy=0x7f4654070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:35.752 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.752+0000 7f46798ba640 10 monclient: shutdown 2024-03-25T19:47:35.752 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.752+0000 7f46798ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:35.752 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.752+0000 7f46798ba640 1 -- 172.21.15.81:0/4090335943 >> v1:172.21.15.81:6789/0 conn(0x7f46740a98f0 legacy=0x7f4674136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:35.752 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.752+0000 7f46798ba640 1 -- 172.21.15.81:0/4090335943 shutdown_connections 2024-03-25T19:47:35.752 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.752+0000 7f46798ba640 1 -- 172.21.15.81:0/4090335943 >> 172.21.15.81:0/4090335943 conn(0x7f46740b7ea0 msgr2=0x7f4674074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:35.753 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.752+0000 7f46798ba640 1 -- 172.21.15.81:0/4090335943 shutdown_connections 2024-03-25T19:47:35.753 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.752+0000 7f46798ba640 1 -- 172.21.15.81:0/4090335943 wait complete. 2024-03-25T19:47:35.753 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:35.752+0000 7f46798ba640 1 librados: shutdown 2024-03-25T19:47:35.763 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:42.571 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:42.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.671+0000 7fd2d5139640 10 monclient: get_monmap_and_config 2024-03-25T19:47:42.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.671+0000 7fd2d5139640 10 monclient: build_initial_monmap 2024-03-25T19:47:42.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.671+0000 7fd2d5139640 10 monclient: monmap: 2024-03-25T19:47:42.672 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:47:42.672 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:42.672 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:47:42.673453+0000 2024-03-25T19:47:42.672 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:47:42.673453+0000 2024-03-25T19:47:42.672 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:42.672 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:42.672 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:47:42.673 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:47:42.673 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:47:42.673 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:42.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.672+0000 7fd2d5139640 1 Processor -- start 2024-03-25T19:47:42.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.672+0000 7fd2d5139640 1 -- start start 2024-03-25T19:47:42.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.672+0000 7fd2d5139640 10 monclient: init 2024-03-25T19:47:42.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.672+0000 7fd2d5139640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:42.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.672+0000 7fd2d5139640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:42.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.672+0000 7fd2d5139640 10 monclient(hunting): picked mon.noname-b con 0x7fd2d00b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:47:42.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.672+0000 7fd2d5139640 10 monclient(hunting): picked mon.noname-a con 0x7fd2d00b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:47:42.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.672+0000 7fd2d5139640 10 monclient(hunting): picked mon.noname-c con 0x7fd2d014be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:47:42.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.672+0000 7fd2d5139640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d00af940 con 0x7fd2d00b1730 2024-03-25T19:47:42.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.672+0000 7fd2d5139640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d00b0a70 con 0x7fd2d014be90 2024-03-25T19:47:42.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.672+0000 7fd2d5139640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d00b0d50 con 0x7fd2d00b4d60 2024-03-25T19:47:42.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.672+0000 7fd2d5139640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:42.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.672+0000 7fd2d5139640 10 monclient(hunting): authenticate will time out at 4297.755567s 2024-03-25T19:47:42.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.673+0000 7fd2ced9d640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fd2d00b4d60 0x7fd2d013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:58714/0 (socket says 172.21.15.81:58714) 2024-03-25T19:47:42.675 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.673+0000 7fd2ced9d640 1 -- 172.21.15.81:0/2350412911 learned_addr learned my addr 172.21.15.81:0/2350412911 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:47:42.685 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:06:05.326 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:42.692 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:06:05.334 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:42.716 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:06:05.358 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:42.733 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.732+0000 7fd2cdd9b640 1 -- 172.21.15.81:0/2350412911 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992615250 0 0) 0x7fd2d00af940 con 0x7fd2d00b1730 2024-03-25T19:47:42.733 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.732+0000 7fd2cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:47:42.733 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.732+0000 7fd2cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:42.733 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.732+0000 7fd2cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:42.733 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.732+0000 7fd2cdd9b640 10 monclient(hunting): my global_id is 4488 2024-03-25T19:47:42.733 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.732+0000 7fd2cdd9b640 1 -- 172.21.15.81:0/2350412911 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2b8003520 con 0x7fd2d00b1730 2024-03-25T19:47:42.738 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:05.372 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:42.739 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.739+0000 7fd2cdd9b640 1 -- 172.21.15.81:0/2350412911 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469649172 0 0) 0x7fd2d00b0a70 con 0x7fd2d014be90 2024-03-25T19:47:42.739 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.739+0000 7fd2cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:47:42.739 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.739+0000 7fd2cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:42.739 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.739+0000 7fd2cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:42.739 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.739+0000 7fd2cdd9b640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:47:42.739 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.739+0000 7fd2cdd9b640 1 -- 172.21.15.81:0/2350412911 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d00af940 con 0x7fd2d014be90 2024-03-25T19:47:42.748 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:05.383 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:42.752 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:06:05.394 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:42.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.756+0000 7fd2cdd9b640 1 -- 172.21.15.81:0/2350412911 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054486513 0 0) 0x7fd2d00b0d50 con 0x7fd2d00b4d60 2024-03-25T19:47:42.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.756+0000 7fd2cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:47:42.758 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.756+0000 7fd2cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:42.758 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.756+0000 7fd2cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:42.758 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.756+0000 7fd2cdd9b640 10 monclient(hunting): my global_id is 4466 2024-03-25T19:47:42.758 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.756+0000 7fd2cdd9b640 1 -- 172.21.15.81:0/2350412911 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d00b0a70 con 0x7fd2d00b4d60 2024-03-25T19:47:42.761 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:06:05.403 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:42.771 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2cdd9b640 1 -- 172.21.15.81:0/2350412911 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3043974703 0 0) 0x7fd2b8003520 con 0x7fd2d00b1730 2024-03-25T19:47:42.771 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2cdd9b640 1 -- 172.21.15.81:0/2350412911 >> v1:172.21.15.81:6790/0 conn(0x7fd2d00b4d60 legacy=0x7fd2d013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:42.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2cdd9b640 1 -- 172.21.15.81:0/2350412911 >> v1:172.21.15.98:6789/0 conn(0x7fd2d014be90 legacy=0x7fd2d014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:42.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2cdd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:47:42.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2cdd9b640 1 monclient: found mon.noname-a 2024-03-25T19:47:42.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:42.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2cdd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:47:42.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2cdd9b640 1 -- 172.21.15.81:0/2350412911 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d00b1030 con 0x7fd2d00b1730 2024-03-25T19:47:42.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2cdd9b640 10 monclient: _finish_auth 0 2024-03-25T19:47:42.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2cdd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:47:42.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:42.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2cdd9b640 1 -- 172.21.15.81:0/2350412911 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd2c4001690 con 0x7fd2d00b1730 2024-03-25T19:47:42.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2d5139640 5 monclient: authenticate success, global_id 4488 2024-03-25T19:47:42.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2d5139640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:42.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:42.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2cdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:42.772 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2cdd9b640 10 monclient: dump: 2024-03-25T19:47:42.773 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:42.773 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:42.773 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:42.773 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:42.773 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:42.773 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:42.773 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:42.773 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:42.773 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:42.773 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:42.773 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.771+0000 7fd2d5139640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:42.782 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:05.416 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:42.785 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:06:05.427 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:42.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.801+0000 7fd2cdd9b640 1 -- 172.21.15.81:0/2350412911 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2c40017f0 con 0x7fd2d00b1730 2024-03-25T19:47:42.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.801+0000 7fd2cdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:42.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.801+0000 7fd2cdd9b640 1 -- 172.21.15.81:0/2350412911 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd2c4001c00 con 0x7fd2d00b1730 2024-03-25T19:47:42.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.801+0000 7fd2d5139640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:42.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.801+0000 7fd2d5139640 4 set_mon_vals no callback set 2024-03-25T19:47:42.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.801+0000 7fd2cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:42.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.801+0000 7fd2cdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:42.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.801+0000 7fd2cdd9b640 10 monclient: dump: 2024-03-25T19:47:42.802 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:42.802 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:42.802 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:42.802 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:42.802 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:42.803 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:42.803 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:42.803 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:42.803 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:42.803 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:42.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.802+0000 7fd2d5139640 10 monclient: shutdown 2024-03-25T19:47:42.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.802+0000 7fd2d5139640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:42.806 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.802+0000 7fd2d5139640 1 -- 172.21.15.81:0/2350412911 >> v1:172.21.15.81:6789/0 conn(0x7fd2d00b1730 legacy=0x7fd2d00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:42.806 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.802+0000 7fd2d5139640 1 -- 172.21.15.81:0/2350412911 shutdown_connections 2024-03-25T19:47:42.806 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.802+0000 7fd2d5139640 1 -- 172.21.15.81:0/2350412911 >> 172.21.15.81:0/2350412911 conn(0x7fd2d00617a0 msgr2=0x7fd2d0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:42.806 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.802+0000 7fd2d5139640 1 -- 172.21.15.81:0/2350412911 shutdown_connections 2024-03-25T19:47:42.806 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.802+0000 7fd2d5139640 1 -- 172.21.15.81:0/2350412911 wait complete. 2024-03-25T19:47:42.806 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.803+0000 7fd2d5139640 10 monclient: build_initial_monmap 2024-03-25T19:47:42.806 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.803+0000 7fd2d5139640 10 monclient: monmap: 2024-03-25T19:47:42.806 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:47:42.806 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:42.806 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:47:42.806 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:47:42.806 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:42.806 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:47:42.807 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:47:42.807 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:47:42.807 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:47:42.807 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:42.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.803+0000 7fd2d5139640 1 librados: starting msgr at 2024-03-25T19:47:42.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.803+0000 7fd2d5139640 1 librados: starting objecter 2024-03-25T19:47:42.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.803+0000 7fd2d5139640 1 Processor -- start 2024-03-25T19:47:42.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.803+0000 7fd2d5139640 1 -- start start 2024-03-25T19:47:42.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.803+0000 7fd2d5139640 1 librados: setting wanted keys 2024-03-25T19:47:42.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.803+0000 7fd2d5139640 1 librados: calling monclient init 2024-03-25T19:47:42.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.803+0000 7fd2d5139640 10 monclient: init 2024-03-25T19:47:42.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.804+0000 7fd2d5139640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:42.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.804+0000 7fd2d5139640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:42.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.804+0000 7fd2d5139640 10 monclient(hunting): picked mon.noname-b con 0x7fd2d00b1730 addr v1:172.21.15.98:6789/0 2024-03-25T19:47:42.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.804+0000 7fd2d5139640 10 monclient(hunting): picked mon.noname-a con 0x7fd2d00b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:47:42.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.804+0000 7fd2d5139640 10 monclient(hunting): picked mon.noname-c con 0x7fd2d014be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:47:42.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.804+0000 7fd2d5139640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d0145910 con 0x7fd2d00b4d60 2024-03-25T19:47:42.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.804+0000 7fd2d5139640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d0135810 con 0x7fd2d00b1730 2024-03-25T19:47:42.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.804+0000 7fd2d5139640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d01453f0 con 0x7fd2d014be90 2024-03-25T19:47:42.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.804+0000 7fd2d5139640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:42.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.804+0000 7fd2d5139640 10 monclient(hunting): authenticate will time out at 4297.887450s 2024-03-25T19:47:42.828 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.828+0000 7fd2cf59e640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fd2d014be90 0x7fd2d0135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:58766/0 (socket says 172.21.15.81:58766) 2024-03-25T19:47:42.828 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.828+0000 7fd2cf59e640 1 -- 172.21.15.81:0/2998553574 learned_addr learned my addr 172.21.15.81:0/2998553574 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:47:42.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.844+0000 7fd2b77fe640 10 client.?.objecter ms_handle_connect 0x7fd2d00b1730 2024-03-25T19:47:42.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.844+0000 7fd2b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:42.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.845+0000 7fd2b77fe640 10 client.?.objecter ms_handle_connect 0x7fd2d014be90 2024-03-25T19:47:42.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.845+0000 7fd2b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:42.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.855+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144088298 0 0) 0x7fd2d0135810 con 0x7fd2d00b1730 2024-03-25T19:47:42.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.855+0000 7fd2b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:42.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.855+0000 7fd2b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:42.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.855+0000 7fd2b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:42.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.855+0000 7fd2b77fe640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:47:42.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.855+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2bc004eb0 con 0x7fd2d00b1730 2024-03-25T19:47:42.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.865+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 454756956 0 0) 0x7fd2bc004eb0 con 0x7fd2d00b1730 2024-03-25T19:47:42.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.865+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2d0135810 con 0x7fd2d00b1730 2024-03-25T19:47:42.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.865+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd2c00018f0 con 0x7fd2d00b1730 2024-03-25T19:47:42.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.865+0000 7fd2b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:42.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.865+0000 7fd2b77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:42.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.865+0000 7fd2b77fe640 10 monclient(hunting): dump: 2024-03-25T19:47:42.866 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:42.866 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:42.866 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:42.866 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:42.866 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:42.866 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:42.866 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:42.866 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:42.867 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:42.867 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:42.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.865+0000 7fd2b77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:42.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.867+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938121170 0 0) 0x7fd2d01453f0 con 0x7fd2d014be90 2024-03-25T19:47:42.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.867+0000 7fd2b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:42.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.867+0000 7fd2b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:42.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.867+0000 7fd2b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:42.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.867+0000 7fd2b77fe640 10 monclient(hunting): my global_id is 4469 2024-03-25T19:47:42.868 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.867+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2bc004eb0 con 0x7fd2d014be90 2024-03-25T19:47:42.869 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.869+0000 7fd2b77fe640 10 client.?.objecter ms_handle_connect 0x7fd2d00b4d60 2024-03-25T19:47:42.869 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.869+0000 7fd2b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:42.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.887+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853080409 0 0) 0x7fd2d0145910 con 0x7fd2d00b4d60 2024-03-25T19:47:42.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.887+0000 7fd2b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:42.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.887+0000 7fd2b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:42.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.887+0000 7fd2b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:42.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.887+0000 7fd2b77fe640 10 monclient(hunting): my global_id is 4491 2024-03-25T19:47:42.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.887+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d01453f0 con 0x7fd2d00b4d60 2024-03-25T19:47:42.889 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.888+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2144020972 0 0) 0x7fd2bc004eb0 con 0x7fd2d014be90 2024-03-25T19:47:42.889 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.888+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2d0145910 con 0x7fd2d014be90 2024-03-25T19:47:42.889 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.888+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd2c8001bd0 con 0x7fd2d014be90 2024-03-25T19:47:42.889 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.888+0000 7fd2b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:42.889 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.888+0000 7fd2b77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:42.889 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.888+0000 7fd2b77fe640 10 monclient(hunting): dump: 2024-03-25T19:47:42.889 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:42.889 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:42.889 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:42.889 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:42.890 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:42.890 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:42.890 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:42.890 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:42.890 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:42.890 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:42.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1272002345 0 0) 0x7fd2d0135810 con 0x7fd2d00b1730 2024-03-25T19:47:42.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 >> v1:172.21.15.81:6790/0 conn(0x7fd2d014be90 legacy=0x7fd2d0135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:42.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 >> v1:172.21.15.81:6789/0 conn(0x7fd2d00b4d60 legacy=0x7fd2d01349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:42.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2b77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:47:42.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2b77fe640 1 monclient: found mon.b 2024-03-25T19:47:42.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:42.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2b77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:42.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d0137bc0 con 0x7fd2d00b1730 2024-03-25T19:47:42.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2b77fe640 10 monclient: _finish_auth 0 2024-03-25T19:47:42.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2b77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:47:42.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:42.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2d5139640 5 monclient: authenticate success, global_id 4483 2024-03-25T19:47:42.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2d5139640 10 monclient: _renew_subs 2024-03-25T19:47:42.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2d5139640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:42.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2d5139640 1 -- 172.21.15.81:0/2998553574 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d01405f0 con 0x7fd2d00b1730 2024-03-25T19:47:42.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2d5139640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:42.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2d5139640 10 monclient: _renew_subs 2024-03-25T19:47:42.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2d5139640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:42.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2d5139640 1 -- 172.21.15.81:0/2998553574 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d01d7da0 con 0x7fd2d00b1730 2024-03-25T19:47:42.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.901+0000 7fd2d5139640 1 librados: init done 2024-03-25T19:47:42.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.902+0000 7fd2d5139640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:42.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.902+0000 7fd2d5139640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:42.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.902+0000 7fd2d5139640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:42.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:42.902+0000 7fd2d5139640 1 -- 172.21.15.81:0/2998553574 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d0145580 con 0x7fd2d00b1730 2024-03-25T19:47:43.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.210+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2c00047e0 con 0x7fd2d00b1730 2024-03-25T19:47:43.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.210+0000 7fd2b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:43.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.210+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd2c0004bf0 con 0x7fd2d00b1730 2024-03-25T19:47:43.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.210+0000 7fd2b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:43.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.210+0000 7fd2b77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:43.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.210+0000 7fd2b77fe640 10 monclient: dump: 2024-03-25T19:47:43.211 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:43.211 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:43.211 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:43.211 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:43.211 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:43.212 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:43.212 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:43.212 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:43.212 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:43.212 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:43.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.210+0000 7fd2cd59a640 4 set_mon_vals no callback set 2024-03-25T19:47:43.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.211+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fd2c0004e10 con 0x7fd2d00b1730 2024-03-25T19:47:43.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.211+0000 7fd2ce59c640 10 monclient: get_auth_request con 0x7fd2bc06e060 auth_method 0 2024-03-25T19:47:43.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.211+0000 7fd2b77fe640 10 client.4483.objecter ms_handle_connect 0x7fd2bc06e060 2024-03-25T19:47:43.217 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.216+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fd2c0054820 con 0x7fd2d00b1730 2024-03-25T19:47:43.217 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.216+0000 7fd2b77fe640 10 client.4483.objecter ms_dispatch 0x7fd2d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:43.217 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.216+0000 7fd2b77fe640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:43.217 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.216+0000 7fd2b77fe640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:43.217 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.216+0000 7fd2b77fe640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-25T19:47:43.217 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.216+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2c0050330 con 0x7fd2d00b1730 2024-03-25T19:47:43.217 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.216+0000 7fd2b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:43.217 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.216+0000 7fd2b77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:43.467 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.466+0000 7fd2d5139640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:43.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.466+0000 7fd2d5139640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:43.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.466+0000 7fd2d5139640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:43.468 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.466+0000 7fd2d5139640 1 -- 172.21.15.81:0/2998553574 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2d00627b0 con 0x7fd2d00b1730 2024-03-25T19:47:43.502 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.502+0000 7fd2b77fe640 1 -- 172.21.15.81:0/2998553574 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fd2c0052260 con 0x7fd2d00b1730 2024-03-25T19:47:43.502 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.502+0000 7fd2b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:43.502 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.502+0000 7fd2b77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:43.503 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:47:43.503 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:43.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.503+0000 7fd2d5139640 10 librados: watch_flush enter 2024-03-25T19:47:43.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.503+0000 7fd2d5139640 10 librados: watch_flush exit 2024-03-25T19:47:43.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.503+0000 7fd2d5139640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-25T19:47:43.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.503+0000 7fd2d5139640 10 client.4483.objecter successfully canceled tick 2024-03-25T19:47:43.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.504+0000 7fd2d5139640 1 -- 172.21.15.81:0/2998553574 >> v1:172.21.15.98:6800/3980829775 conn(0x7fd2bc06e060 legacy=0x7fd2bc070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:43.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.504+0000 7fd2d5139640 10 monclient: shutdown 2024-03-25T19:47:43.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.504+0000 7fd2d5139640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:43.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.504+0000 7fd2d5139640 1 -- 172.21.15.81:0/2998553574 >> v1:172.21.15.98:6789/0 conn(0x7fd2d00b1730 legacy=0x7fd2d01d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:43.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.504+0000 7fd2d5139640 1 -- 172.21.15.81:0/2998553574 shutdown_connections 2024-03-25T19:47:43.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.504+0000 7fd2d5139640 1 -- 172.21.15.81:0/2998553574 >> 172.21.15.81:0/2998553574 conn(0x7fd2d00617a0 msgr2=0x7fd2d013c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:43.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.504+0000 7fd2d5139640 1 -- 172.21.15.81:0/2998553574 shutdown_connections 2024-03-25T19:47:43.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.504+0000 7fd2d5139640 1 -- 172.21.15.81:0/2998553574 wait complete. 2024-03-25T19:47:43.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:43.504+0000 7fd2d5139640 1 librados: shutdown 2024-03-25T19:47:43.517 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:50.324 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.441+0000 7f5b6eff4640 10 monclient: get_monmap_and_config 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.441+0000 7f5b6eff4640 10 monclient: build_initial_monmap 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.441+0000 7f5b6eff4640 10 monclient: monmap: 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:47:50.442342+0000 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:47:50.442342+0000 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:50.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.442+0000 7f5b6eff4640 1 Processor -- start 2024-03-25T19:47:50.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.442+0000 7f5b6eff4640 1 -- start start 2024-03-25T19:47:50.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.442+0000 7f5b6eff4640 10 monclient: init 2024-03-25T19:47:50.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.442+0000 7f5b6eff4640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:50.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.442+0000 7f5b6eff4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:50.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.442+0000 7f5b6eff4640 10 monclient(hunting): picked mon.noname-b con 0x7f5b680b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:47:50.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.442+0000 7f5b6eff4640 10 monclient(hunting): picked mon.noname-a con 0x7f5b680b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:47:50.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.442+0000 7f5b6eff4640 10 monclient(hunting): picked mon.noname-c con 0x7f5b6814be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:47:50.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.442+0000 7f5b6eff4640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b680af940 con 0x7f5b680b1730 2024-03-25T19:47:50.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.442+0000 7f5b6eff4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b680b0a70 con 0x7f5b6814be90 2024-03-25T19:47:50.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.442+0000 7f5b6eff4640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b680b0d50 con 0x7f5b680b4d60 2024-03-25T19:47:50.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.442+0000 7f5b6eff4640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:50.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.442+0000 7f5b6eff4640 10 monclient(hunting): authenticate will time out at 4305.524699s 2024-03-25T19:47:50.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.457+0000 7f5b6d591640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5b6814be90 0x7f5b6814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:53410/0 (socket says 172.21.15.81:53410) 2024-03-25T19:47:50.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.457+0000 7f5b6d591640 1 -- 172.21.15.81:0/4197683113 learned_addr learned my addr 172.21.15.81:0/4197683113 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:47:50.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.465+0000 7f5b677fe640 1 -- 172.21.15.81:0/4197683113 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046027049 0 0) 0x7f5b680b0a70 con 0x7f5b6814be90 2024-03-25T19:47:50.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.465+0000 7f5b677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:50.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.465+0000 7f5b677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:50.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.465+0000 7f5b677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:50.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.465+0000 7f5b677fe640 10 monclient(hunting): my global_id is 4486 2024-03-25T19:47:50.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.465+0000 7f5b677fe640 1 -- 172.21.15.81:0/4197683113 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b58003520 con 0x7f5b6814be90 2024-03-25T19:47:50.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.479+0000 7f5b6cd90640 1 -- 172.21.15.81:0/4197683113 >> v1:172.21.15.81:6790/0 conn(0x7f5b680b4d60 legacy=0x7f5b6813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T19:47:50.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.479+0000 7f5b6cd90640 1 -- 172.21.15.81:0/4197683113 >> v1:172.21.15.81:6790/0 conn(0x7f5b680b4d60 legacy=0x7f5b6813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:47:50.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.479+0000 7f5b6cd90640 1 --1- 172.21.15.81:0/4197683113 >> v1:172.21.15.81:6790/0 conn(0x7f5b680b4d60 0x7f5b6813f660 :-1 s=OPENED pgs=42596 cs=1 l=1).handle_message read tag failed 2024-03-25T19:47:50.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.479+0000 7f5b6cd90640 1 --1- 172.21.15.81:0/4197683113 >> v1:172.21.15.81:6790/0 conn(0x7f5b680b4d60 0x7f5b6813f660 :-1 s=OPENED pgs=42596 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:47:50.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.479+0000 7f5b677fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.81:6790/0 2024-03-25T19:47:50.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.481+0000 7f5b677fe640 1 -- 172.21.15.81:0/4197683113 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 480726319 0 0) 0x7f5b58003520 con 0x7f5b6814be90 2024-03-25T19:47:50.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.481+0000 7f5b677fe640 1 -- 172.21.15.81:0/4197683113 >> v1:172.21.15.81:6790/0 conn(0x7f5b680b4d60 legacy=0x7f5b6813f660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:47:50.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.481+0000 7f5b677fe640 1 -- 172.21.15.81:0/4197683113 >> v1:172.21.15.81:6789/0 conn(0x7f5b680b1730 legacy=0x7f5b680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:50.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.481+0000 7f5b677fe640 10 monclient: _finish_hunting 0 2024-03-25T19:47:50.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.481+0000 7f5b677fe640 1 monclient: found mon.noname-c 2024-03-25T19:47:50.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.481+0000 7f5b677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:50.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.482+0000 7f5b677fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:47:50.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.482+0000 7f5b677fe640 1 -- 172.21.15.81:0/4197683113 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b680b1030 con 0x7f5b6814be90 2024-03-25T19:47:50.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.482+0000 7f5b677fe640 10 monclient: _finish_auth 0 2024-03-25T19:47:50.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.482+0000 7f5b677fe640 10 monclient: _check_auth_tickets 2024-03-25T19:47:50.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.482+0000 7f5b677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:50.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.482+0000 7f5b677fe640 1 -- 172.21.15.81:0/4197683113 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f5b60001a10 con 0x7f5b6814be90 2024-03-25T19:47:50.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.482+0000 7f5b677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:50.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.482+0000 7f5b677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:50.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.482+0000 7f5b677fe640 10 monclient: dump: 2024-03-25T19:47:50.483 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:50.483 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:50.483 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:50.483 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:50.483 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:50.483 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:50.483 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:50.483 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:50.483 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:50.483 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:50.483 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.482+0000 7f5b6eff4640 5 monclient: authenticate success, global_id 4486 2024-03-25T19:47:50.483 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.482+0000 7f5b6eff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:50.484 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.485+0000 7f5b677fe640 1 -- 172.21.15.81:0/4197683113 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b60001b70 con 0x7f5b6814be90 2024-03-25T19:47:50.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.485+0000 7f5b677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:50.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.485+0000 7f5b6eff4640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:50.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.485+0000 7f5b6eff4640 4 set_mon_vals no callback set 2024-03-25T19:47:50.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.486+0000 7f5b6eff4640 10 monclient: shutdown 2024-03-25T19:47:50.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.486+0000 7f5b6eff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:50.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.486+0000 7f5b6eff4640 1 -- 172.21.15.81:0/4197683113 >> v1:172.21.15.98:6789/0 conn(0x7f5b6814be90 legacy=0x7f5b6814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:50.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.486+0000 7f5b6eff4640 1 -- 172.21.15.81:0/4197683113 shutdown_connections 2024-03-25T19:47:50.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.486+0000 7f5b6eff4640 1 -- 172.21.15.81:0/4197683113 >> 172.21.15.81:0/4197683113 conn(0x7f5b680617a0 msgr2=0x7f5b68061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:50.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.486+0000 7f5b6eff4640 1 -- 172.21.15.81:0/4197683113 shutdown_connections 2024-03-25T19:47:50.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.486+0000 7f5b6eff4640 1 -- 172.21.15.81:0/4197683113 wait complete. 2024-03-25T19:47:50.487 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.487+0000 7f5b6eff4640 10 monclient: build_initial_monmap 2024-03-25T19:47:50.487 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.487+0000 7f5b6eff4640 10 monclient: monmap: 2024-03-25T19:47:50.487 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:47:50.487 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:50.487 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:47:50.487 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:47:50.487 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:50.487 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:47:50.487 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:47:50.487 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:47:50.487 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:47:50.487 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:50.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.487+0000 7f5b6eff4640 1 librados: starting msgr at 2024-03-25T19:47:50.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.487+0000 7f5b6eff4640 1 librados: starting objecter 2024-03-25T19:47:50.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.487+0000 7f5b6eff4640 1 Processor -- start 2024-03-25T19:47:50.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.487+0000 7f5b6eff4640 1 -- start start 2024-03-25T19:47:50.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.487+0000 7f5b6eff4640 1 librados: setting wanted keys 2024-03-25T19:47:50.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.487+0000 7f5b6eff4640 1 librados: calling monclient init 2024-03-25T19:47:50.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.487+0000 7f5b6eff4640 10 monclient: init 2024-03-25T19:47:50.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.488+0000 7f5b6eff4640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:50.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.488+0000 7f5b6eff4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:50.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.488+0000 7f5b6eff4640 10 monclient(hunting): picked mon.noname-a con 0x7f5b680b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:47:50.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.488+0000 7f5b6eff4640 10 monclient(hunting): picked mon.noname-b con 0x7f5b680b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:47:50.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.488+0000 7f5b6eff4640 10 monclient(hunting): picked mon.noname-c con 0x7f5b6814be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:47:50.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.488+0000 7f5b6eff4640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b681dbc40 con 0x7f5b680b1730 2024-03-25T19:47:50.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.488+0000 7f5b6eff4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b681dad30 con 0x7f5b680b4d60 2024-03-25T19:47:50.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.488+0000 7f5b6eff4640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b681403f0 con 0x7f5b6814be90 2024-03-25T19:47:50.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.488+0000 7f5b6eff4640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:50.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.488+0000 7f5b6eff4640 10 monclient(hunting): authenticate will time out at 4305.570230s 2024-03-25T19:47:50.499 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.500+0000 7f5b67fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f5b680b4d60 0x7f5b6813b4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:53434/0 (socket says 172.21.15.81:53434) 2024-03-25T19:47:50.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.500+0000 7f5b67fff640 1 -- 172.21.15.81:0/1727004427 learned_addr learned my addr 172.21.15.81:0/1727004427 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:47:50.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.501+0000 7f5b657fa640 10 client.?.objecter ms_handle_connect 0x7f5b680b4d60 2024-03-25T19:47:50.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.501+0000 7f5b657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:50.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.511+0000 7f5b657fa640 10 client.?.objecter ms_handle_connect 0x7f5b6814be90 2024-03-25T19:47:50.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.511+0000 7f5b657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:50.519 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.519+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909207203 0 0) 0x7f5b681403f0 con 0x7f5b6814be90 2024-03-25T19:47:50.519 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.519+0000 7f5b657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:50.519 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.519+0000 7f5b657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:50.519 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.519+0000 7f5b657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:50.519 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.519+0000 7f5b657fa640 10 monclient(hunting): my global_id is 4472 2024-03-25T19:47:50.519 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.519+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b4c004eb0 con 0x7f5b6814be90 2024-03-25T19:47:50.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.523+0000 7f5b657fa640 10 client.?.objecter ms_handle_connect 0x7f5b680b1730 2024-03-25T19:47:50.523 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.524+0000 7f5b657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:50.525 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.525+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902004862 0 0) 0x7f5b681dad30 con 0x7f5b680b4d60 2024-03-25T19:47:50.525 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.525+0000 7f5b657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:50.525 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.525+0000 7f5b657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:50.525 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.525+0000 7f5b657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:50.525 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.525+0000 7f5b657fa640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:47:50.525 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.525+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b681403f0 con 0x7f5b680b4d60 2024-03-25T19:47:50.534 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.535+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796079676 0 0) 0x7f5b681dbc40 con 0x7f5b680b1730 2024-03-25T19:47:50.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.535+0000 7f5b657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:50.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.535+0000 7f5b657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:50.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.535+0000 7f5b657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:50.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.535+0000 7f5b657fa640 10 monclient(hunting): my global_id is 4497 2024-03-25T19:47:50.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.535+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b681dad30 con 0x7f5b680b1730 2024-03-25T19:47:50.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.539+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3277658944 0 0) 0x7f5b4c004eb0 con 0x7f5b6814be90 2024-03-25T19:47:50.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.539+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b681dbc40 con 0x7f5b6814be90 2024-03-25T19:47:50.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.539+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f5b60001e80 con 0x7f5b6814be90 2024-03-25T19:47:50.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.539+0000 7f5b657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:50.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.539+0000 7f5b657fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:50.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.539+0000 7f5b657fa640 10 monclient(hunting): dump: 2024-03-25T19:47:50.539 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:50.539 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:50.539 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:50.539 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:50.539 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:50.539 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:50.539 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:50.539 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:50.540 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:50.540 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:50.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.539+0000 7f5b657fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:50.541 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.542+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4059264553 0 0) 0x7f5b681dad30 con 0x7f5b680b1730 2024-03-25T19:47:50.542 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.542+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b4c004eb0 con 0x7f5b680b1730 2024-03-25T19:47:50.542 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.542+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f5b5c0018a0 con 0x7f5b680b1730 2024-03-25T19:47:50.542 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.542+0000 7f5b657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:50.542 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.542+0000 7f5b657fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:50.542 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.542+0000 7f5b657fa640 10 monclient(hunting): dump: 2024-03-25T19:47:50.542 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:50.542 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:50.542 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:50.542 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:50.542 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:50.542 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:50.542 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:50.542 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:50.542 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:50.543 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:50.544 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.544+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1996610621 0 0) 0x7f5b681403f0 con 0x7f5b680b4d60 2024-03-25T19:47:50.544 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.544+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b681dad30 con 0x7f5b680b4d60 2024-03-25T19:47:50.544 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.544+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f5b540019b0 con 0x7f5b680b4d60 2024-03-25T19:47:50.544 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.544+0000 7f5b657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:50.544 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.544+0000 7f5b657fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:50.544 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.544+0000 7f5b657fa640 10 monclient(hunting): dump: 2024-03-25T19:47:50.544 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:50.545 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:50.545 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:50.545 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:50.545 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:50.545 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:50.545 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:50.545 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:50.545 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:50.545 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:50.547 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3109212119 0 0) 0x7f5b4c004eb0 con 0x7f5b680b1730 2024-03-25T19:47:50.547 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 >> v1:172.21.15.81:6790/0 conn(0x7f5b6814be90 legacy=0x7f5b681e2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:50.547 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 >> v1:172.21.15.98:6789/0 conn(0x7f5b680b4d60 legacy=0x7f5b6813b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:50.547 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b657fa640 10 monclient: _finish_hunting 0 2024-03-25T19:47:50.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b657fa640 1 monclient: found mon.a 2024-03-25T19:47:50.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:50.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b657fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:47:50.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b68140580 con 0x7f5b680b1730 2024-03-25T19:47:50.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b657fa640 10 monclient: _finish_auth 0 2024-03-25T19:47:50.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b657fa640 10 monclient: _check_auth_tickets 2024-03-25T19:47:50.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:50.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b6eff4640 5 monclient: authenticate success, global_id 4497 2024-03-25T19:47:50.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b6eff4640 10 monclient: _renew_subs 2024-03-25T19:47:50.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b6eff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:47:50.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b6eff4640 1 -- 172.21.15.81:0/1727004427 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6813bc00 con 0x7f5b680b1730 2024-03-25T19:47:50.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b6eff4640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:50.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b6eff4640 10 monclient: _renew_subs 2024-03-25T19:47:50.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b6eff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:47:50.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b6eff4640 1 -- 172.21.15.81:0/1727004427 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b681391c0 con 0x7f5b680b1730 2024-03-25T19:47:50.549 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.547+0000 7f5b6eff4640 1 librados: init done 2024-03-25T19:47:50.549 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.549+0000 7f5b6eff4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:50.549 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.549+0000 7f5b6eff4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:50.549 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.549+0000 7f5b6eff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:47:50.549 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.549+0000 7f5b6eff4640 1 -- 172.21.15.81:0/1727004427 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b681401f0 con 0x7f5b680b1730 2024-03-25T19:47:50.836 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.837+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b5c001d70 con 0x7f5b680b1730 2024-03-25T19:47:50.836 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.837+0000 7f5b657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:50.837 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.837+0000 7f5b66ffd640 4 set_mon_vals no callback set 2024-03-25T19:47:50.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.846+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f5b5c003810 con 0x7f5b680b1730 2024-03-25T19:47:50.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.846+0000 7f5b657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:50.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.846+0000 7f5b657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:50.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.846+0000 7f5b657fa640 10 monclient: dump: 2024-03-25T19:47:50.846 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:50.846 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:50.846 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:50.846 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:50.846 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:50.846 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:50.846 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:50.846 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:50.846 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:50.846 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:50.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.847+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f5b5c0039f0 con 0x7f5b680b1730 2024-03-25T19:47:50.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.847+0000 7f5b67fff640 10 monclient: get_auth_request con 0x7f5b4c06e110 auth_method 0 2024-03-25T19:47:50.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.848+0000 7f5b657fa640 10 client.4497.objecter ms_handle_connect 0x7f5b4c06e110 2024-03-25T19:47:50.857 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.858+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f5b5c0547e0 con 0x7f5b680b1730 2024-03-25T19:47:50.857 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.858+0000 7f5b657fa640 10 client.4497.objecter ms_dispatch 0x7f5b68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:50.857 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.858+0000 7f5b657fa640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:50.857 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.858+0000 7f5b657fa640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:50.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.858+0000 7f5b657fa640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-25T19:47:50.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.858+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b5c050120 con 0x7f5b680b1730 2024-03-25T19:47:50.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.858+0000 7f5b657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:50.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:50.858+0000 7f5b657fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:51.103 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.103+0000 7f5b6eff4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:51.103 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.103+0000 7f5b6eff4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:51.103 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.103+0000 7f5b6eff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:47:51.103 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.103+0000 7f5b6eff4640 1 -- 172.21.15.81:0/1727004427 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b680b2270 con 0x7f5b680b1730 2024-03-25T19:47:51.131 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.131+0000 7f5b657fa640 1 -- 172.21.15.81:0/1727004427 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f5b5c052220 con 0x7f5b680b1730 2024-03-25T19:47:51.131 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.131+0000 7f5b657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:51.131 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.131+0000 7f5b657fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:51.131 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:47:51.131 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:51.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.134+0000 7f5b6eff4640 10 librados: watch_flush enter 2024-03-25T19:47:51.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.134+0000 7f5b6eff4640 10 librados: watch_flush exit 2024-03-25T19:47:51.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.134+0000 7f5b6eff4640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-25T19:47:51.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.134+0000 7f5b6eff4640 10 client.4497.objecter successfully canceled tick 2024-03-25T19:47:51.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.134+0000 7f5b6eff4640 1 -- 172.21.15.81:0/1727004427 >> v1:172.21.15.98:6800/3980829775 conn(0x7f5b4c06e110 legacy=0x7f5b4c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:51.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.134+0000 7f5b6eff4640 10 monclient: shutdown 2024-03-25T19:47:51.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.134+0000 7f5b6eff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:51.135 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.134+0000 7f5b6eff4640 1 -- 172.21.15.81:0/1727004427 >> v1:172.21.15.81:6789/0 conn(0x7f5b680b1730 legacy=0x7f5b681db530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:51.135 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.134+0000 7f5b6eff4640 1 -- 172.21.15.81:0/1727004427 shutdown_connections 2024-03-25T19:47:51.135 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.134+0000 7f5b6eff4640 1 -- 172.21.15.81:0/1727004427 >> 172.21.15.81:0/1727004427 conn(0x7f5b680617a0 msgr2=0x7f5b6813c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:51.135 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.134+0000 7f5b6eff4640 1 -- 172.21.15.81:0/1727004427 shutdown_connections 2024-03-25T19:47:51.135 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.134+0000 7f5b6eff4640 1 -- 172.21.15.81:0/1727004427 wait complete. 2024-03-25T19:47:51.135 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:51.134+0000 7f5b6eff4640 1 librados: shutdown 2024-03-25T19:47:51.146 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:57.953 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:58.068 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.069+0000 7f11ca2b3640 10 monclient: get_monmap_and_config 2024-03-25T19:47:58.068 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.069+0000 7f11ca2b3640 10 monclient: build_initial_monmap 2024-03-25T19:47:58.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.069+0000 7f11ca2b3640 10 monclient: monmap: 2024-03-25T19:47:58.069 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:47:58.069 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:58.069 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:47:58.070007+0000 2024-03-25T19:47:58.069 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:47:58.070007+0000 2024-03-25T19:47:58.069 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:58.069 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:58.069 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:47:58.069 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:47:58.069 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:47:58.069 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:58.070 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.069+0000 7f11ca2b3640 1 Processor -- start 2024-03-25T19:47:58.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.069+0000 7f11ca2b3640 1 -- start start 2024-03-25T19:47:58.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.069+0000 7f11ca2b3640 10 monclient: init 2024-03-25T19:47:58.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.070+0000 7f11ca2b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:58.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.070+0000 7f11ca2b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:47:58.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.070+0000 7f11ca2b3640 10 monclient(hunting): picked mon.noname-c con 0x7f11c40a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:47:58.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.070+0000 7f11ca2b3640 10 monclient(hunting): picked mon.noname-b con 0x7f11c413f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:47:58.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.070+0000 7f11ca2b3640 10 monclient(hunting): picked mon.noname-a con 0x7f11c41430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:47:58.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.070+0000 7f11ca2b3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c4142520 con 0x7f11c41430c0 2024-03-25T19:47:58.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.070+0000 7f11ca2b3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c40b28a0 con 0x7f11c40a98f0 2024-03-25T19:47:58.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.070+0000 7f11ca2b3640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c40b2a30 con 0x7f11c413f710 2024-03-25T19:47:58.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.070+0000 7f11ca2b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:58.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.070+0000 7f11ca2b3640 10 monclient(hunting): authenticate will time out at 4313.152505s 2024-03-25T19:47:58.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.072+0000 7f11c37fe640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f11c413f710 0x7f11c41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:46398/0 (socket says 172.21.15.81:46398) 2024-03-25T19:47:58.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.072+0000 7f11c37fe640 1 -- 172.21.15.81:0/667216387 learned_addr learned my addr 172.21.15.81:0/667216387 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:47:58.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.086+0000 7f11c2ffd640 1 -- 172.21.15.81:0/667216387 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838813512 0 0) 0x7f11c40b2a30 con 0x7f11c413f710 2024-03-25T19:47:58.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.086+0000 7f11c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:58.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.086+0000 7f11c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:58.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.086+0000 7f11c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:58.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.086+0000 7f11c2ffd640 10 monclient(hunting): my global_id is 4475 2024-03-25T19:47:58.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.086+0000 7f11c2ffd640 1 -- 172.21.15.81:0/667216387 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11ac003520 con 0x7f11c413f710 2024-03-25T19:47:58.099 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.099+0000 7f11c2ffd640 1 -- 172.21.15.81:0/667216387 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327560591 0 0) 0x7f11c4142520 con 0x7f11c41430c0 2024-03-25T19:47:58.099 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.099+0000 7f11c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:58.099 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.099+0000 7f11c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:58.099 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.099+0000 7f11c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:58.099 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.099+0000 7f11c2ffd640 10 monclient(hunting): my global_id is 4500 2024-03-25T19:47:58.099 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.099+0000 7f11c2ffd640 1 -- 172.21.15.81:0/667216387 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c40b2a30 con 0x7f11c41430c0 2024-03-25T19:47:58.101 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.102+0000 7f11c2ffd640 1 -- 172.21.15.81:0/667216387 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 338319312 0 0) 0x7f11ac003520 con 0x7f11c413f710 2024-03-25T19:47:58.101 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.102+0000 7f11c2ffd640 1 -- 172.21.15.81:0/667216387 >> v1:172.21.15.98:6789/0 conn(0x7f11c40a98f0 legacy=0x7f11c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:58.102 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.102+0000 7f11c2ffd640 1 -- 172.21.15.81:0/667216387 >> v1:172.21.15.81:6789/0 conn(0x7f11c41430c0 legacy=0x7f11c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:58.102 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.102+0000 7f11c2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:58.102 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.102+0000 7f11c2ffd640 1 monclient: found mon.noname-b 2024-03-25T19:47:58.102 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.102+0000 7f11c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:58.102 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.102+0000 7f11c2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:47:58.102 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.102+0000 7f11c2ffd640 1 -- 172.21.15.81:0/667216387 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c40b44c0 con 0x7f11c413f710 2024-03-25T19:47:58.102 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.102+0000 7f11c2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:58.102 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.102+0000 7f11c2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:58.102 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.102+0000 7f11c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:58.102 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.102+0000 7f11c2ffd640 1 -- 172.21.15.81:0/667216387 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f11b8001690 con 0x7f11c413f710 2024-03-25T19:47:58.102 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.102+0000 7f11c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:58.102 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.102+0000 7f11c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:58.103 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.102+0000 7f11c2ffd640 10 monclient: dump: 2024-03-25T19:47:58.103 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:58.103 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:58.103 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:58.103 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:58.103 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:58.103 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:58.103 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:58.103 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:58.103 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:58.103 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:58.103 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.102+0000 7f11ca2b3640 5 monclient: authenticate success, global_id 4475 2024-03-25T19:47:58.103 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.102+0000 7f11ca2b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:58.124 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.124+0000 7f11c2ffd640 1 -- 172.21.15.81:0/667216387 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11b80017f0 con 0x7f11c413f710 2024-03-25T19:47:58.124 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.124+0000 7f11c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:58.124 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.124+0000 7f11c2ffd640 1 -- 172.21.15.81:0/667216387 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f11b8001c00 con 0x7f11c413f710 2024-03-25T19:47:58.125 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.124+0000 7f11ca2b3640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:58.125 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.124+0000 7f11ca2b3640 4 set_mon_vals no callback set 2024-03-25T19:47:58.125 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.124+0000 7f11c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:58.125 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.124+0000 7f11c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:58.125 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.124+0000 7f11c2ffd640 10 monclient: dump: 2024-03-25T19:47:58.125 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:58.125 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:58.125 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:58.125 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:58.125 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:58.125 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:58.125 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:58.126 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:58.126 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:58.126 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:58.127 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.125+0000 7f11ca2b3640 10 monclient: shutdown 2024-03-25T19:47:58.127 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.125+0000 7f11ca2b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:58.127 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.125+0000 7f11ca2b3640 1 -- 172.21.15.81:0/667216387 >> v1:172.21.15.81:6790/0 conn(0x7f11c413f710 legacy=0x7f11c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:58.127 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.125+0000 7f11ca2b3640 1 -- 172.21.15.81:0/667216387 shutdown_connections 2024-03-25T19:47:58.127 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.125+0000 7f11ca2b3640 1 -- 172.21.15.81:0/667216387 >> 172.21.15.81:0/667216387 conn(0x7f11c40b7ea0 msgr2=0x7f11c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:58.127 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.125+0000 7f11ca2b3640 1 -- 172.21.15.81:0/667216387 shutdown_connections 2024-03-25T19:47:58.127 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.125+0000 7f11ca2b3640 1 -- 172.21.15.81:0/667216387 wait complete. 2024-03-25T19:47:58.127 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.126+0000 7f11ca2b3640 10 monclient: build_initial_monmap 2024-03-25T19:47:58.127 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.126+0000 7f11ca2b3640 10 monclient: monmap: 2024-03-25T19:47:58.128 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:47:58.128 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:58.128 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:47:58.128 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:47:58.128 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:58.128 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:47:58.128 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:47:58.128 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:47:58.128 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:47:58.128 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:58.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.127+0000 7f11ca2b3640 1 librados: starting msgr at 2024-03-25T19:47:58.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.127+0000 7f11ca2b3640 1 librados: starting objecter 2024-03-25T19:47:58.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.127+0000 7f11ca2b3640 1 Processor -- start 2024-03-25T19:47:58.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.127+0000 7f11ca2b3640 1 -- start start 2024-03-25T19:47:58.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.127+0000 7f11ca2b3640 1 librados: setting wanted keys 2024-03-25T19:47:58.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.127+0000 7f11ca2b3640 1 librados: calling monclient init 2024-03-25T19:47:58.128 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.127+0000 7f11ca2b3640 10 monclient: init 2024-03-25T19:47:58.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.128+0000 7f11ca2b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:58.130 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.128+0000 7f11ca2b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:47:58.130 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.128+0000 7f11ca2b3640 10 monclient(hunting): picked mon.noname-b con 0x7f11c40a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:47:58.130 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.128+0000 7f11ca2b3640 10 monclient(hunting): picked mon.noname-c con 0x7f11c413f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:47:58.130 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.128+0000 7f11ca2b3640 10 monclient(hunting): picked mon.noname-a con 0x7f11c41430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:47:58.130 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.128+0000 7f11ca2b3640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c4139a20 con 0x7f11c41430c0 2024-03-25T19:47:58.130 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.128+0000 7f11ca2b3640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c4070610 con 0x7f11c40a98f0 2024-03-25T19:47:58.130 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.128+0000 7f11ca2b3640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c40ac3f0 con 0x7f11c413f710 2024-03-25T19:47:58.130 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.128+0000 7f11ca2b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:58.130 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.128+0000 7f11ca2b3640 10 monclient(hunting): authenticate will time out at 4313.210541s 2024-03-25T19:47:58.144 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.145+0000 7f11c3fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f11c40a98f0 0x7f11c40715f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:37394/0 (socket says 172.21.15.81:37394) 2024-03-25T19:47:58.144 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.145+0000 7f11c3fff640 1 -- 172.21.15.81:0/2684508740 learned_addr learned my addr 172.21.15.81:0/2684508740 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:47:58.148 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.149+0000 7f11c0ff9640 10 client.?.objecter ms_handle_connect 0x7f11c40a98f0 2024-03-25T19:47:58.148 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.149+0000 7f11c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:58.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.153+0000 7f11c0ff9640 10 client.?.objecter ms_handle_connect 0x7f11c41430c0 2024-03-25T19:47:58.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.153+0000 7f11c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:58.153 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.153+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688108105 0 0) 0x7f11c4070610 con 0x7f11c40a98f0 2024-03-25T19:47:58.153 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.153+0000 7f11c0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:58.153 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.153+0000 7f11c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:58.153 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.153+0000 7f11c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:58.153 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.153+0000 7f11c0ff9640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:47:58.153 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.153+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11b0004eb0 con 0x7f11c40a98f0 2024-03-25T19:47:58.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.163+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219207034 0 0) 0x7f11c4139a20 con 0x7f11c41430c0 2024-03-25T19:47:58.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.163+0000 7f11c0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:58.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.163+0000 7f11c0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:58.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.163+0000 7f11c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:58.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.163+0000 7f11c0ff9640 10 monclient(hunting): my global_id is 4503 2024-03-25T19:47:58.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.163+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c4070610 con 0x7f11c41430c0 2024-03-25T19:47:58.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.164+0000 7f11c0ff9640 10 client.?.objecter ms_handle_connect 0x7f11c413f710 2024-03-25T19:47:58.164 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.164+0000 7f11c0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:58.173 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.173+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4138211924 0 0) 0x7f11b0004eb0 con 0x7f11c40a98f0 2024-03-25T19:47:58.173 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.173+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11c4139a20 con 0x7f11c40a98f0 2024-03-25T19:47:58.173 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.173+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f11b40019d0 con 0x7f11c40a98f0 2024-03-25T19:47:58.173 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.173+0000 7f11c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:58.173 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.173+0000 7f11c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:58.173 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.173+0000 7f11c0ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:58.173 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:58.173 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:58.173 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:58.174 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:58.174 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:58.174 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:58.174 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:58.174 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:58.174 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:58.174 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:58.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.173+0000 7f11c0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:58.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.177+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 872054795 0 0) 0x7f11c4070610 con 0x7f11c41430c0 2024-03-25T19:47:58.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.177+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11b0004eb0 con 0x7f11c41430c0 2024-03-25T19:47:58.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.178+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f11bc001bd0 con 0x7f11c41430c0 2024-03-25T19:47:58.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.178+0000 7f11c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:58.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.178+0000 7f11c0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:58.178 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.178+0000 7f11c0ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:58.178 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:58.178 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:58.178 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:58.178 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:58.178 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:58.178 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:58.178 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:58.179 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:58.179 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:58.179 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:58.183 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3777868223 0 0) 0x7f11c4139a20 con 0x7f11c40a98f0 2024-03-25T19:47:58.183 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 >> v1:172.21.15.81:6790/0 conn(0x7f11c413f710 legacy=0x7f11c406ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:58.183 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 >> v1:172.21.15.81:6789/0 conn(0x7f11c41430c0 legacy=0x7f11c41e68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:58.183 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11c0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:47:58.183 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11c0ff9640 1 monclient: found mon.b 2024-03-25T19:47:58.183 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:58.183 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11c0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:58.183 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c40ac6d0 con 0x7f11c40a98f0 2024-03-25T19:47:58.183 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11c0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:47:58.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11c0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:47:58.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:58.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11ca2b3640 5 monclient: authenticate success, global_id 4495 2024-03-25T19:47:58.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11ca2b3640 10 monclient: _renew_subs 2024-03-25T19:47:58.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11ca2b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:58.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11ca2b3640 1 -- 172.21.15.81:0/2684508740 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c40ac9b0 con 0x7f11c40a98f0 2024-03-25T19:47:58.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11ca2b3640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:58.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11ca2b3640 10 monclient: _renew_subs 2024-03-25T19:47:58.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11ca2b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:58.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11ca2b3640 1 -- 172.21.15.81:0/2684508740 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c4070ec0 con 0x7f11c40a98f0 2024-03-25T19:47:58.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.183+0000 7f11ca2b3640 1 librados: init done 2024-03-25T19:47:58.185 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.184+0000 7f11ca2b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:58.185 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.184+0000 7f11ca2b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:58.185 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.184+0000 7f11ca2b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:58.185 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.184+0000 7f11ca2b3640 1 -- 172.21.15.81:0/2684508740 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c4139810 con 0x7f11c40a98f0 2024-03-25T19:47:58.462 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.462+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11b4003610 con 0x7f11c40a98f0 2024-03-25T19:47:58.462 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.462+0000 7f11c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:58.462 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.462+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f11b4003a20 con 0x7f11c40a98f0 2024-03-25T19:47:58.462 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.462+0000 7f11c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:58.462 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.462+0000 7f11c0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:58.462 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.462+0000 7f11c0ff9640 10 monclient: dump: 2024-03-25T19:47:58.462 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:47:58.463 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:47:58.463 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:58.463 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:47:58.463 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:58.463 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:47:58.463 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:47:58.463 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:47:58.463 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:47:58.463 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:47:58.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.462+0000 7f11c27fc640 4 set_mon_vals no callback set 2024-03-25T19:47:58.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.463+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f11b4003c40 con 0x7f11c40a98f0 2024-03-25T19:47:58.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.463+0000 7f11c37fe640 10 monclient: get_auth_request con 0x7f11b006de90 auth_method 0 2024-03-25T19:47:58.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.464+0000 7f11c0ff9640 10 client.4495.objecter ms_handle_connect 0x7f11b006de90 2024-03-25T19:47:58.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.469+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f11b40547f0 con 0x7f11c40a98f0 2024-03-25T19:47:58.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.469+0000 7f11c0ff9640 10 client.4495.objecter ms_dispatch 0x7f11c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:58.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.469+0000 7f11c0ff9640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:58.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.469+0000 7f11c0ff9640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:58.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.469+0000 7f11c0ff9640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-25T19:47:58.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.469+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11b4050300 con 0x7f11c40a98f0 2024-03-25T19:47:58.469 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.469+0000 7f11c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:58.470 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.469+0000 7f11c0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:58.707 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.707+0000 7f11ca2b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:58.707 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.707+0000 7f11ca2b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:58.707 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.707+0000 7f11ca2b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:47:58.707 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.707+0000 7f11ca2b3640 1 -- 172.21.15.81:0/2684508740 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11c40627b0 con 0x7f11c40a98f0 2024-03-25T19:47:58.733 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.733+0000 7f11c0ff9640 1 -- 172.21.15.81:0/2684508740 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f11b4052230 con 0x7f11c40a98f0 2024-03-25T19:47:58.733 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.733+0000 7f11c0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:58.733 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.733+0000 7f11c0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:58.733 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:47:58.733 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:58.735 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.735+0000 7f11ca2b3640 10 librados: watch_flush enter 2024-03-25T19:47:58.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.735+0000 7f11ca2b3640 10 librados: watch_flush exit 2024-03-25T19:47:58.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.735+0000 7f11ca2b3640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-25T19:47:58.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.735+0000 7f11ca2b3640 10 client.4495.objecter successfully canceled tick 2024-03-25T19:47:58.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.735+0000 7f11ca2b3640 1 -- 172.21.15.81:0/2684508740 >> v1:172.21.15.98:6800/3980829775 conn(0x7f11b006de90 legacy=0x7f11b0070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:58.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.735+0000 7f11ca2b3640 10 monclient: shutdown 2024-03-25T19:47:58.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.735+0000 7f11ca2b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:58.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.735+0000 7f11ca2b3640 1 -- 172.21.15.81:0/2684508740 >> v1:172.21.15.98:6789/0 conn(0x7f11c40a98f0 legacy=0x7f11c40715f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:58.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.735+0000 7f11ca2b3640 1 -- 172.21.15.81:0/2684508740 shutdown_connections 2024-03-25T19:47:58.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.735+0000 7f11ca2b3640 1 -- 172.21.15.81:0/2684508740 >> 172.21.15.81:0/2684508740 conn(0x7f11c40b7ea0 msgr2=0x7f11c413d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:58.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.736+0000 7f11ca2b3640 1 -- 172.21.15.81:0/2684508740 shutdown_connections 2024-03-25T19:47:58.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.736+0000 7f11ca2b3640 1 -- 172.21.15.81:0/2684508740 wait complete. 2024-03-25T19:47:58.736 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:47:58.736+0000 7f11ca2b3640 1 librados: shutdown 2024-03-25T19:47:58.748 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:02.662 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:06:25.303 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:02.675 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:06:25.317 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:02.713 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:06:25.355 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:02.724 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:25.359 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:02.729 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:25.363 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:02.748 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:06:25.389 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:02.756 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:06:25.398 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:02.768 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:25.402 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:02.787 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:06:25.429 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:05.554 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:05.671 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.671+0000 7f82be4e6640 10 monclient: get_monmap_and_config 2024-03-25T19:48:05.671 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.672+0000 7f82be4e6640 10 monclient: build_initial_monmap 2024-03-25T19:48:05.671 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.672+0000 7f82be4e6640 10 monclient: monmap: 2024-03-25T19:48:05.671 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:48:05.672 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:05.672 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:48:05.672933+0000 2024-03-25T19:48:05.672 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:48:05.672933+0000 2024-03-25T19:48:05.672 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:05.672 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:05.672 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:48:05.672 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:48:05.672 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:48:05.672 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:05.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.672+0000 7f82be4e6640 1 Processor -- start 2024-03-25T19:48:05.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.673+0000 7f82be4e6640 1 -- start start 2024-03-25T19:48:05.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.673+0000 7f82be4e6640 10 monclient: init 2024-03-25T19:48:05.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.673+0000 7f82be4e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:05.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.673+0000 7f82be4e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:05.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.673+0000 7f82be4e6640 10 monclient(hunting): picked mon.noname-a con 0x7f82b80a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:48:05.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.673+0000 7f82be4e6640 10 monclient(hunting): picked mon.noname-b con 0x7f82b813f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:48:05.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.673+0000 7f82be4e6640 10 monclient(hunting): picked mon.noname-c con 0x7f82b81430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:48:05.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.673+0000 7f82be4e6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b8142520 con 0x7f82b80a98f0 2024-03-25T19:48:05.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.673+0000 7f82be4e6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b80b28a0 con 0x7f82b81430c0 2024-03-25T19:48:05.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.673+0000 7f82be4e6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b80b2a30 con 0x7f82b813f710 2024-03-25T19:48:05.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.673+0000 7f82be4e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:05.674 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.673+0000 7f82be4e6640 10 monclient(hunting): authenticate will time out at 4320.755523s 2024-03-25T19:48:05.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.678+0000 7f82b7fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f82b80a98f0 0x7f82b80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:53230/0 (socket says 172.21.15.81:53230) 2024-03-25T19:48:05.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.678+0000 7f82b7fff640 1 -- 172.21.15.81:0/3273813006 learned_addr learned my addr 172.21.15.81:0/3273813006 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:48:05.692 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.693+0000 7f82b6ffd640 1 -- 172.21.15.81:0/3273813006 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150916543 0 0) 0x7f82b8142520 con 0x7f82b80a98f0 2024-03-25T19:48:05.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.693+0000 7f82b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:05.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.693+0000 7f82b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:05.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.693+0000 7f82b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:05.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.693+0000 7f82b6ffd640 10 monclient(hunting): my global_id is 4506 2024-03-25T19:48:05.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.693+0000 7f82b6ffd640 1 -- 172.21.15.81:0/3273813006 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a0003520 con 0x7f82b80a98f0 2024-03-25T19:48:05.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.693+0000 7f82b6ffd640 1 -- 172.21.15.81:0/3273813006 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680506563 0 0) 0x7f82b80b28a0 con 0x7f82b81430c0 2024-03-25T19:48:05.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.693+0000 7f82b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:05.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.693+0000 7f82b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:05.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.693+0000 7f82b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:05.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.693+0000 7f82b6ffd640 10 monclient(hunting): my global_id is 4498 2024-03-25T19:48:05.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.693+0000 7f82b6ffd640 1 -- 172.21.15.81:0/3273813006 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b8142520 con 0x7f82b81430c0 2024-03-25T19:48:05.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.702+0000 7f82b6ffd640 1 -- 172.21.15.81:0/3273813006 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 594653382 0 0) 0x7f82b8142520 con 0x7f82b81430c0 2024-03-25T19:48:05.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.702+0000 7f82b6ffd640 1 -- 172.21.15.81:0/3273813006 >> v1:172.21.15.81:6790/0 conn(0x7f82b813f710 legacy=0x7f82b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:05.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.702+0000 7f82b6ffd640 1 -- 172.21.15.81:0/3273813006 >> v1:172.21.15.81:6789/0 conn(0x7f82b80a98f0 legacy=0x7f82b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:05.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.702+0000 7f82b6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:48:05.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.702+0000 7f82b6ffd640 1 monclient: found mon.noname-c 2024-03-25T19:48:05.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.702+0000 7f82b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:05.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.702+0000 7f82b6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:48:05.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.702+0000 7f82b6ffd640 1 -- 172.21.15.81:0/3273813006 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b80b44c0 con 0x7f82b81430c0 2024-03-25T19:48:05.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.702+0000 7f82b6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:48:05.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.702+0000 7f82b6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:48:05.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.702+0000 7f82b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:05.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.702+0000 7f82b6ffd640 1 -- 172.21.15.81:0/3273813006 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f82b00019f0 con 0x7f82b81430c0 2024-03-25T19:48:05.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.702+0000 7f82b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:05.703 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.702+0000 7f82b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:05.703 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.702+0000 7f82b6ffd640 10 monclient: dump: 2024-03-25T19:48:05.703 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:05.703 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:05.703 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:05.703 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:05.703 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:05.703 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:05.703 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:05.703 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:05.703 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:05.703 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:05.703 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.702+0000 7f82be4e6640 5 monclient: authenticate success, global_id 4498 2024-03-25T19:48:05.703 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.702+0000 7f82be4e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:05.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.723+0000 7f82b6ffd640 1 -- 172.21.15.81:0/3273813006 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82b0001b50 con 0x7f82b81430c0 2024-03-25T19:48:05.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.723+0000 7f82b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:05.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.723+0000 7f82b6ffd640 1 -- 172.21.15.81:0/3273813006 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f82b0001f20 con 0x7f82b81430c0 2024-03-25T19:48:05.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.723+0000 7f82be4e6640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:05.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.723+0000 7f82be4e6640 4 set_mon_vals no callback set 2024-03-25T19:48:05.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.723+0000 7f82b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:05.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.723+0000 7f82b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:05.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.723+0000 7f82b6ffd640 10 monclient: dump: 2024-03-25T19:48:05.723 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:05.724 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:05.724 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:05.724 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:05.724 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:05.724 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:05.724 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:05.724 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:05.724 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:05.724 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:05.725 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.724+0000 7f82be4e6640 10 monclient: shutdown 2024-03-25T19:48:05.725 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.724+0000 7f82be4e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:05.726 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.724+0000 7f82be4e6640 1 -- 172.21.15.81:0/3273813006 >> v1:172.21.15.98:6789/0 conn(0x7f82b81430c0 legacy=0x7f82b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:05.726 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.724+0000 7f82be4e6640 1 -- 172.21.15.81:0/3273813006 shutdown_connections 2024-03-25T19:48:05.726 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.724+0000 7f82be4e6640 1 -- 172.21.15.81:0/3273813006 >> 172.21.15.81:0/3273813006 conn(0x7f82b80b7ea0 msgr2=0x7f82b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:05.726 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.724+0000 7f82be4e6640 1 -- 172.21.15.81:0/3273813006 shutdown_connections 2024-03-25T19:48:05.726 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.724+0000 7f82be4e6640 1 -- 172.21.15.81:0/3273813006 wait complete. 2024-03-25T19:48:05.726 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.725+0000 7f82be4e6640 10 monclient: build_initial_monmap 2024-03-25T19:48:05.726 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.725+0000 7f82be4e6640 10 monclient: monmap: 2024-03-25T19:48:05.726 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:48:05.726 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:05.726 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:48:05.726 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:48:05.726 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:05.726 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:48:05.726 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:48:05.727 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:48:05.727 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:48:05.727 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:05.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.725+0000 7f82be4e6640 1 librados: starting msgr at 2024-03-25T19:48:05.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.725+0000 7f82be4e6640 1 librados: starting objecter 2024-03-25T19:48:05.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.726+0000 7f82be4e6640 1 Processor -- start 2024-03-25T19:48:05.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.726+0000 7f82be4e6640 1 -- start start 2024-03-25T19:48:05.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.726+0000 7f82be4e6640 1 librados: setting wanted keys 2024-03-25T19:48:05.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.726+0000 7f82be4e6640 1 librados: calling monclient init 2024-03-25T19:48:05.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.726+0000 7f82be4e6640 10 monclient: init 2024-03-25T19:48:05.728 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.727+0000 7f82be4e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:05.728 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.727+0000 7f82be4e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:05.728 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.727+0000 7f82be4e6640 10 monclient(hunting): picked mon.noname-b con 0x7f82b80a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:48:05.728 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.727+0000 7f82be4e6640 10 monclient(hunting): picked mon.noname-a con 0x7f82b813f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:48:05.728 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.727+0000 7f82be4e6640 10 monclient(hunting): picked mon.noname-c con 0x7f82b81430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:48:05.728 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.727+0000 7f82be4e6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b8138f80 con 0x7f82b813f710 2024-03-25T19:48:05.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.727+0000 7f82be4e6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b812fac0 con 0x7f82b80a98f0 2024-03-25T19:48:05.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.727+0000 7f82be4e6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b812fda0 con 0x7f82b81430c0 2024-03-25T19:48:05.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.727+0000 7f82be4e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:05.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.727+0000 7f82be4e6640 10 monclient(hunting): authenticate will time out at 4320.809440s 2024-03-25T19:48:05.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.741+0000 7f82bca83640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f82b81430c0 0x7f82b81e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:50952/0 (socket says 172.21.15.81:50952) 2024-03-25T19:48:05.741 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.741+0000 7f82bca83640 1 -- 172.21.15.81:0/4036368234 learned_addr learned my addr 172.21.15.81:0/4036368234 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:48:05.745 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.746+0000 7f82b4ff9640 10 client.?.objecter ms_handle_connect 0x7f82b81430c0 2024-03-25T19:48:05.745 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.746+0000 7f82b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:05.745 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.746+0000 7f82b4ff9640 10 client.?.objecter ms_handle_connect 0x7f82b813f710 2024-03-25T19:48:05.746 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.746+0000 7f82b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:05.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.749+0000 7f82b4ff9640 10 client.?.objecter ms_handle_connect 0x7f82b80a98f0 2024-03-25T19:48:05.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.749+0000 7f82b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:05.749 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.750+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921255 0 0) 0x7f82b8138f80 con 0x7f82b813f710 2024-03-25T19:48:05.750 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.750+0000 7f82b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:48:05.750 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.750+0000 7f82b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:05.750 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.750+0000 7f82b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:05.750 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.750+0000 7f82b4ff9640 10 monclient(hunting): my global_id is 4509 2024-03-25T19:48:05.750 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.750+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82a4004eb0 con 0x7f82b813f710 2024-03-25T19:48:05.754 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.755+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155877326 0 0) 0x7f82b812fac0 con 0x7f82b80a98f0 2024-03-25T19:48:05.755 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.755+0000 7f82b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:48:05.755 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.755+0000 7f82b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:05.755 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.755+0000 7f82b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:05.755 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.755+0000 7f82b4ff9640 10 monclient(hunting): my global_id is 4501 2024-03-25T19:48:05.755 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.755+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b8138f80 con 0x7f82b80a98f0 2024-03-25T19:48:05.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.761+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292593419 0 0) 0x7f82b812fda0 con 0x7f82b81430c0 2024-03-25T19:48:05.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.761+0000 7f82b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:48:05.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.761+0000 7f82b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:05.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.761+0000 7f82b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:05.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.761+0000 7f82b4ff9640 10 monclient(hunting): my global_id is 4484 2024-03-25T19:48:05.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.761+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b812fac0 con 0x7f82b81430c0 2024-03-25T19:48:05.763 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.763+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1119229115 0 0) 0x7f82a4004eb0 con 0x7f82b813f710 2024-03-25T19:48:05.763 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.763+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82b812fda0 con 0x7f82b813f710 2024-03-25T19:48:05.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.763+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f82ac0018c0 con 0x7f82b813f710 2024-03-25T19:48:05.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.763+0000 7f82b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:05.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.763+0000 7f82b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:05.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.763+0000 7f82b4ff9640 10 monclient(hunting): dump: 2024-03-25T19:48:05.764 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:05.764 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:05.764 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:05.764 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:05.764 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:05.764 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:05.764 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:05.764 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:05.764 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:05.764 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:05.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.763+0000 7f82b4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:05.773 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.773+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1459179654 0 0) 0x7f82b8138f80 con 0x7f82b80a98f0 2024-03-25T19:48:05.773 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.773+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82a4004eb0 con 0x7f82b80a98f0 2024-03-25T19:48:05.773 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.773+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f82a80018d0 con 0x7f82b80a98f0 2024-03-25T19:48:05.773 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.773+0000 7f82b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:05.773 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.773+0000 7f82b4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:05.773 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.773+0000 7f82b4ff9640 10 monclient(hunting): dump: 2024-03-25T19:48:05.773 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:05.774 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:05.774 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:05.774 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:05.774 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:05.774 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:05.774 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:05.774 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:05.774 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:05.774 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:05.775 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3379078317 0 0) 0x7f82b812fda0 con 0x7f82b813f710 2024-03-25T19:48:05.775 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 >> v1:172.21.15.81:6790/0 conn(0x7f82b81430c0 legacy=0x7f82b81e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:05.776 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 >> v1:172.21.15.98:6789/0 conn(0x7f82b80a98f0 legacy=0x7f82b812e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:05.776 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82b4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:48:05.776 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82b4ff9640 1 monclient: found mon.a 2024-03-25T19:48:05.776 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:05.776 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82b4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:05.776 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b8130080 con 0x7f82b813f710 2024-03-25T19:48:05.776 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82b4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:48:05.776 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82b4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:48:05.776 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:05.776 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82be4e6640 5 monclient: authenticate success, global_id 4509 2024-03-25T19:48:05.776 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82be4e6640 10 monclient: _renew_subs 2024-03-25T19:48:05.776 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82be4e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:05.776 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82be4e6640 1 -- 172.21.15.81:0/4036368234 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b81401f0 con 0x7f82b813f710 2024-03-25T19:48:05.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82be4e6640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:05.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82be4e6640 10 monclient: _renew_subs 2024-03-25T19:48:05.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82be4e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:05.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82be4e6640 1 -- 172.21.15.81:0/4036368234 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b813b1f0 con 0x7f82b813f710 2024-03-25T19:48:05.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.775+0000 7f82be4e6640 1 librados: init done 2024-03-25T19:48:05.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.776+0000 7f82be4e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:05.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.776+0000 7f82be4e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:05.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.776+0000 7f82be4e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:05.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:05.776+0000 7f82be4e6640 1 -- 172.21.15.81:0/4036368234 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b80b3ef0 con 0x7f82b813f710 2024-03-25T19:48:06.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.076+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82ac0035f0 con 0x7f82b813f710 2024-03-25T19:48:06.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.076+0000 7f82b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:06.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.076+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f82ac003900 con 0x7f82b813f710 2024-03-25T19:48:06.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.076+0000 7f82b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:06.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.076+0000 7f82b4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:06.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.076+0000 7f82b4ff9640 10 monclient: dump: 2024-03-25T19:48:06.077 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:06.077 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:06.077 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:06.077 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:06.077 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:06.077 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:06.077 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:06.077 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:06.077 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:06.078 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:06.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.076+0000 7f82b5ffb640 4 set_mon_vals no callback set 2024-03-25T19:48:06.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.077+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f82ac0198d0 con 0x7f82b813f710 2024-03-25T19:48:06.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.078+0000 7f82b7fff640 10 monclient: get_auth_request con 0x7f82a406df20 auth_method 0 2024-03-25T19:48:06.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.078+0000 7f82b4ff9640 10 client.4509.objecter ms_handle_connect 0x7f82a406df20 2024-03-25T19:48:06.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.083+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f82ac0535e0 con 0x7f82b813f710 2024-03-25T19:48:06.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.083+0000 7f82b4ff9640 10 client.4509.objecter ms_dispatch 0x7f82b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:06.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.083+0000 7f82b4ff9640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:06.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.083+0000 7f82b4ff9640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:06.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.083+0000 7f82b4ff9640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-25T19:48:06.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.083+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82ac04f0f0 con 0x7f82b813f710 2024-03-25T19:48:06.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.083+0000 7f82b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:06.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.083+0000 7f82b4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:06.333 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.333+0000 7f82be4e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:06.333 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.333+0000 7f82be4e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:06.333 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.333+0000 7f82be4e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:06.333 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.333+0000 7f82be4e6640 1 -- 172.21.15.81:0/4036368234 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82b812ef90 con 0x7f82b813f710 2024-03-25T19:48:06.361 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.361+0000 7f82b4ff9640 1 -- 172.21.15.81:0/4036368234 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f82b812ef90 con 0x7f82b813f710 2024-03-25T19:48:06.361 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.362+0000 7f82b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:06.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.362+0000 7f82b4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:06.362 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:48:06.362 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:06.364 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.364+0000 7f82be4e6640 10 librados: watch_flush enter 2024-03-25T19:48:06.364 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.364+0000 7f82be4e6640 10 librados: watch_flush exit 2024-03-25T19:48:06.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.364+0000 7f82be4e6640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-25T19:48:06.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.364+0000 7f82be4e6640 10 client.4509.objecter successfully canceled tick 2024-03-25T19:48:06.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.364+0000 7f82be4e6640 1 -- 172.21.15.81:0/4036368234 >> v1:172.21.15.98:6800/3980829775 conn(0x7f82a406df20 legacy=0x7f82a40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:06.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.364+0000 7f82be4e6640 10 monclient: shutdown 2024-03-25T19:48:06.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.364+0000 7f82be4e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:06.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.364+0000 7f82be4e6640 1 -- 172.21.15.81:0/4036368234 >> v1:172.21.15.81:6789/0 conn(0x7f82b813f710 legacy=0x7f82b81df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:06.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.364+0000 7f82be4e6640 1 -- 172.21.15.81:0/4036368234 shutdown_connections 2024-03-25T19:48:06.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.364+0000 7f82be4e6640 1 -- 172.21.15.81:0/4036368234 >> 172.21.15.81:0/4036368234 conn(0x7f82b80b7ea0 msgr2=0x7f82b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:06.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.364+0000 7f82be4e6640 1 -- 172.21.15.81:0/4036368234 shutdown_connections 2024-03-25T19:48:06.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.364+0000 7f82be4e6640 1 -- 172.21.15.81:0/4036368234 wait complete. 2024-03-25T19:48:06.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:06.364+0000 7f82be4e6640 1 librados: shutdown 2024-03-25T19:48:06.376 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:13.183 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:13.302 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.302+0000 7fc42cd82640 10 monclient: get_monmap_and_config 2024-03-25T19:48:13.302 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.302+0000 7fc42cd82640 10 monclient: build_initial_monmap 2024-03-25T19:48:13.302 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.302+0000 7fc42cd82640 10 monclient: monmap: 2024-03-25T19:48:13.302 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:48:13.302 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:13.302 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:48:13.303646+0000 2024-03-25T19:48:13.302 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:48:13.303646+0000 2024-03-25T19:48:13.303 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:13.303 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:13.303 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:48:13.303 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:48:13.303 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:48:13.303 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:13.304 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.303+0000 7fc42cd82640 1 Processor -- start 2024-03-25T19:48:13.304 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.303+0000 7fc42cd82640 1 -- start start 2024-03-25T19:48:13.304 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.303+0000 7fc42cd82640 10 monclient: init 2024-03-25T19:48:13.304 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.304+0000 7fc42cd82640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:13.304 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.304+0000 7fc42cd82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:13.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.304+0000 7fc42cd82640 10 monclient(hunting): picked mon.noname-a con 0x7fc428061830 addr v1:172.21.15.81:6789/0 2024-03-25T19:48:13.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.304+0000 7fc42cd82640 10 monclient(hunting): picked mon.noname-c con 0x7fc42806e740 addr v1:172.21.15.98:6789/0 2024-03-25T19:48:13.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.304+0000 7fc42cd82640 10 monclient(hunting): picked mon.noname-b con 0x7fc4280ae280 addr v1:172.21.15.81:6790/0 2024-03-25T19:48:13.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.304+0000 7fc42cd82640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4280ae830 con 0x7fc428061830 2024-03-25T19:48:13.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.304+0000 7fc42cd82640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc42806f720 con 0x7fc42806e740 2024-03-25T19:48:13.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.304+0000 7fc42cd82640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4280ada80 con 0x7fc4280ae280 2024-03-25T19:48:13.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.304+0000 7fc42cd82640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:13.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.304+0000 7fc42cd82640 10 monclient(hunting): authenticate will time out at 4328.386507s 2024-03-25T19:48:13.312 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.312+0000 7fc42659c640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc428061830 0x7fc42813f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:49240/0 (socket says 172.21.15.81:49240) 2024-03-25T19:48:13.312 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.312+0000 7fc42659c640 1 -- 172.21.15.81:0/3519017623 learned_addr learned my addr 172.21.15.81:0/3519017623 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:48:13.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.330+0000 7fc425d9b640 1 -- 172.21.15.81:0/3519017623 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734186156 0 0) 0x7fc4280ae830 con 0x7fc428061830 2024-03-25T19:48:13.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.331+0000 7fc425d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:48:13.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.331+0000 7fc425d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:13.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.331+0000 7fc425d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:13.331 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.331+0000 7fc425d9b640 10 monclient(hunting): my global_id is 4512 2024-03-25T19:48:13.331 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.331+0000 7fc425d9b640 1 -- 172.21.15.81:0/3519017623 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc414003520 con 0x7fc428061830 2024-03-25T19:48:13.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.336+0000 7fc425d9b640 1 -- 172.21.15.81:0/3519017623 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136443225 0 0) 0x7fc4280ada80 con 0x7fc4280ae280 2024-03-25T19:48:13.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.336+0000 7fc425d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:48:13.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.336+0000 7fc425d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:13.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.336+0000 7fc425d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:13.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.336+0000 7fc425d9b640 10 monclient(hunting): my global_id is 4487 2024-03-25T19:48:13.336 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.336+0000 7fc425d9b640 1 -- 172.21.15.81:0/3519017623 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4280ae830 con 0x7fc4280ae280 2024-03-25T19:48:13.337 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.337+0000 7fc425d9b640 1 -- 172.21.15.81:0/3519017623 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060056238 0 0) 0x7fc42806f720 con 0x7fc42806e740 2024-03-25T19:48:13.337 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.337+0000 7fc425d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:48:13.337 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.337+0000 7fc425d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:13.337 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.337+0000 7fc425d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:13.337 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.337+0000 7fc425d9b640 10 monclient(hunting): my global_id is 4504 2024-03-25T19:48:13.337 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.337+0000 7fc425d9b640 1 -- 172.21.15.81:0/3519017623 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4280ada80 con 0x7fc42806e740 2024-03-25T19:48:13.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.342+0000 7fc425d9b640 1 -- 172.21.15.81:0/3519017623 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1554498797 0 0) 0x7fc414003520 con 0x7fc428061830 2024-03-25T19:48:13.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.342+0000 7fc425d9b640 1 -- 172.21.15.81:0/3519017623 >> v1:172.21.15.81:6790/0 conn(0x7fc4280ae280 legacy=0x7fc428145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:13.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.342+0000 7fc425d9b640 1 -- 172.21.15.81:0/3519017623 >> v1:172.21.15.98:6789/0 conn(0x7fc42806e740 legacy=0x7fc428145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:13.343 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.342+0000 7fc425d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:48:13.343 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.342+0000 7fc425d9b640 1 monclient: found mon.noname-a 2024-03-25T19:48:13.343 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.342+0000 7fc425d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:13.343 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.342+0000 7fc425d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:48:13.343 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.342+0000 7fc425d9b640 1 -- 172.21.15.81:0/3519017623 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42813fcd0 con 0x7fc428061830 2024-03-25T19:48:13.343 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.342+0000 7fc425d9b640 10 monclient: _finish_auth 0 2024-03-25T19:48:13.343 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.342+0000 7fc425d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:48:13.343 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.342+0000 7fc425d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:13.343 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.342+0000 7fc425d9b640 1 -- 172.21.15.81:0/3519017623 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fc418001690 con 0x7fc428061830 2024-03-25T19:48:13.343 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.342+0000 7fc425d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:13.343 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.342+0000 7fc425d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:13.343 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.342+0000 7fc425d9b640 10 monclient: dump: 2024-03-25T19:48:13.343 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:13.343 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:13.344 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:13.344 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:13.344 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:13.344 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:13.344 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:13.344 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:13.344 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:13.344 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:13.344 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.342+0000 7fc42cd82640 5 monclient: authenticate success, global_id 4512 2024-03-25T19:48:13.344 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.342+0000 7fc42cd82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:13.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.362+0000 7fc425d9b640 1 -- 172.21.15.81:0/3519017623 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4180017f0 con 0x7fc428061830 2024-03-25T19:48:13.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.362+0000 7fc425d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:13.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.362+0000 7fc425d9b640 1 -- 172.21.15.81:0/3519017623 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fc418001c00 con 0x7fc428061830 2024-03-25T19:48:13.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.362+0000 7fc425d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:13.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.362+0000 7fc425d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:13.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.362+0000 7fc425d9b640 10 monclient: dump: 2024-03-25T19:48:13.362 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:13.362 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:13.362 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:13.363 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:13.363 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:13.363 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:13.363 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:13.363 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:13.363 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:13.363 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:13.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.362+0000 7fc42cd82640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:13.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.362+0000 7fc42cd82640 4 set_mon_vals no callback set 2024-03-25T19:48:13.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.363+0000 7fc42cd82640 10 monclient: shutdown 2024-03-25T19:48:13.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.363+0000 7fc42cd82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:13.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.363+0000 7fc42cd82640 1 -- 172.21.15.81:0/3519017623 >> v1:172.21.15.81:6789/0 conn(0x7fc428061830 legacy=0x7fc42813f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:13.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.363+0000 7fc42cd82640 1 -- 172.21.15.81:0/3519017623 shutdown_connections 2024-03-25T19:48:13.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.363+0000 7fc42cd82640 1 -- 172.21.15.81:0/3519017623 >> 172.21.15.81:0/3519017623 conn(0x7fc4280b0ef0 msgr2=0x7fc4280b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:13.365 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.363+0000 7fc42cd82640 1 -- 172.21.15.81:0/3519017623 shutdown_connections 2024-03-25T19:48:13.366 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.363+0000 7fc42cd82640 1 -- 172.21.15.81:0/3519017623 wait complete. 2024-03-25T19:48:13.366 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.364+0000 7fc42cd82640 10 monclient: build_initial_monmap 2024-03-25T19:48:13.366 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.364+0000 7fc42cd82640 10 monclient: monmap: 2024-03-25T19:48:13.366 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:48:13.366 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:13.366 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:48:13.366 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:48:13.366 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:13.366 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:48:13.366 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:48:13.366 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:48:13.366 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:48:13.366 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:13.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.364+0000 7fc42cd82640 1 librados: starting msgr at 2024-03-25T19:48:13.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.364+0000 7fc42cd82640 1 librados: starting objecter 2024-03-25T19:48:13.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.364+0000 7fc42cd82640 1 Processor -- start 2024-03-25T19:48:13.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.365+0000 7fc42cd82640 1 -- start start 2024-03-25T19:48:13.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.365+0000 7fc42cd82640 1 librados: setting wanted keys 2024-03-25T19:48:13.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.365+0000 7fc42cd82640 1 librados: calling monclient init 2024-03-25T19:48:13.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.365+0000 7fc42cd82640 10 monclient: init 2024-03-25T19:48:13.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.365+0000 7fc42cd82640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:13.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.365+0000 7fc42cd82640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:13.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.365+0000 7fc42cd82640 10 monclient(hunting): picked mon.noname-b con 0x7fc428061830 addr v1:172.21.15.98:6789/0 2024-03-25T19:48:13.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.365+0000 7fc42cd82640 10 monclient(hunting): picked mon.noname-a con 0x7fc4280ae280 addr v1:172.21.15.81:6789/0 2024-03-25T19:48:13.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.365+0000 7fc42cd82640 10 monclient(hunting): picked mon.noname-c con 0x7fc428131d80 addr v1:172.21.15.81:6790/0 2024-03-25T19:48:13.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.365+0000 7fc42cd82640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc428141790 con 0x7fc4280ae280 2024-03-25T19:48:13.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.365+0000 7fc42cd82640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4281461e0 con 0x7fc428061830 2024-03-25T19:48:13.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.365+0000 7fc42cd82640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc428145e20 con 0x7fc428131d80 2024-03-25T19:48:13.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.365+0000 7fc42cd82640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:13.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.365+0000 7fc42cd82640 10 monclient(hunting): authenticate will time out at 4328.447492s 2024-03-25T19:48:13.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.381+0000 7fc42659c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc428061830 0x7fc428131670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:36478/0 (socket says 172.21.15.81:36478) 2024-03-25T19:48:13.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.381+0000 7fc42659c640 1 -- 172.21.15.81:0/1652813164 learned_addr learned my addr 172.21.15.81:0/1652813164 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:48:13.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.382+0000 7fc41effd640 10 client.?.objecter ms_handle_connect 0x7fc428061830 2024-03-25T19:48:13.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.382+0000 7fc41effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:13.388 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.389+0000 7fc41effd640 10 client.?.objecter ms_handle_connect 0x7fc4280ae280 2024-03-25T19:48:13.388 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.389+0000 7fc41effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:13.389 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.390+0000 7fc41effd640 10 client.?.objecter ms_handle_connect 0x7fc428131d80 2024-03-25T19:48:13.389 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.390+0000 7fc41effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:13.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.408+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108992242 0 0) 0x7fc4281461e0 con 0x7fc428061830 2024-03-25T19:48:13.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.408+0000 7fc41effd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:13.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.408+0000 7fc41effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:13.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.408+0000 7fc41effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:13.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.408+0000 7fc41effd640 10 monclient(hunting): my global_id is 4507 2024-03-25T19:48:13.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.408+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc408004eb0 con 0x7fc428061830 2024-03-25T19:48:13.409 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.409+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076757267 0 0) 0x7fc428141790 con 0x7fc4280ae280 2024-03-25T19:48:13.409 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.409+0000 7fc41effd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:13.409 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.409+0000 7fc41effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:13.409 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.409+0000 7fc41effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:13.409 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.409+0000 7fc41effd640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:48:13.409 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.409+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4281461e0 con 0x7fc4280ae280 2024-03-25T19:48:13.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.410+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916874411 0 0) 0x7fc428145e20 con 0x7fc428131d80 2024-03-25T19:48:13.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.410+0000 7fc41effd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:13.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.410+0000 7fc41effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:13.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.410+0000 7fc41effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:13.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.410+0000 7fc41effd640 10 monclient(hunting): my global_id is 4490 2024-03-25T19:48:13.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.410+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc428141790 con 0x7fc428131d80 2024-03-25T19:48:13.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.423+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3361064657 0 0) 0x7fc4281461e0 con 0x7fc4280ae280 2024-03-25T19:48:13.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.423+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc428145e20 con 0x7fc4280ae280 2024-03-25T19:48:13.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.423+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fc410001870 con 0x7fc4280ae280 2024-03-25T19:48:13.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.423+0000 7fc41effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:13.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.423+0000 7fc41effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:13.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.423+0000 7fc41effd640 10 monclient(hunting): dump: 2024-03-25T19:48:13.423 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:13.423 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:13.423 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:13.424 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:13.424 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:13.424 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:13.424 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:13.424 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:13.424 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:13.424 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:13.424 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.423+0000 7fc41effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:13.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2474161317 0 0) 0x7fc428145e20 con 0x7fc4280ae280 2024-03-25T19:48:13.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 >> v1:172.21.15.81:6790/0 conn(0x7fc428131d80 legacy=0x7fc4281e2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:13.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 >> v1:172.21.15.98:6789/0 conn(0x7fc428061830 legacy=0x7fc428131670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:13.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc41effd640 10 monclient: _finish_hunting 0 2024-03-25T19:48:13.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc41effd640 1 monclient: found mon.a 2024-03-25T19:48:13.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc41effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:13.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc41effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:13.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc428145fb0 con 0x7fc4280ae280 2024-03-25T19:48:13.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc41effd640 10 monclient: _finish_auth 0 2024-03-25T19:48:13.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc41effd640 10 monclient: _check_auth_tickets 2024-03-25T19:48:13.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc41effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:13.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc42cd82640 5 monclient: authenticate success, global_id 4515 2024-03-25T19:48:13.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc42cd82640 10 monclient: _renew_subs 2024-03-25T19:48:13.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc42cd82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:13.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc42cd82640 1 -- 172.21.15.81:0/1652813164 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4280b3470 con 0x7fc4280ae280 2024-03-25T19:48:13.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc42cd82640 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:13.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc42cd82640 10 monclient: _renew_subs 2024-03-25T19:48:13.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc42cd82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:13.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc42cd82640 1 -- 172.21.15.81:0/1652813164 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4281dae40 con 0x7fc4280ae280 2024-03-25T19:48:13.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.433+0000 7fc42cd82640 1 librados: init done 2024-03-25T19:48:13.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.434+0000 7fc42cd82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:13.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.434+0000 7fc42cd82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:13.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.434+0000 7fc42cd82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:13.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.434+0000 7fc42cd82640 1 -- 172.21.15.81:0/1652813164 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4280627b0 con 0x7fc4280ae280 2024-03-25T19:48:13.759 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.759+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc410003610 con 0x7fc4280ae280 2024-03-25T19:48:13.759 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.760+0000 7fc41effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:13.759 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.760+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fc410001d80 con 0x7fc4280ae280 2024-03-25T19:48:13.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.760+0000 7fc41effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:13.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.760+0000 7fc41effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:13.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.760+0000 7fc41effd640 10 monclient: dump: 2024-03-25T19:48:13.760 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:13.760 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:13.760 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:13.760 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:13.760 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:13.760 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:13.760 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:13.760 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:13.760 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:13.760 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:13.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.760+0000 7fc42559a640 4 set_mon_vals no callback set 2024-03-25T19:48:13.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.760+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fc410003b20 con 0x7fc4280ae280 2024-03-25T19:48:13.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.761+0000 7fc42659c640 10 monclient: get_auth_request con 0x7fc40806de50 auth_method 0 2024-03-25T19:48:13.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.762+0000 7fc41effd640 10 client.4515.objecter ms_handle_connect 0x7fc40806de50 2024-03-25T19:48:13.769 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.769+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fc410053590 con 0x7fc4280ae280 2024-03-25T19:48:13.769 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.769+0000 7fc41effd640 10 client.4515.objecter ms_dispatch 0x7fc428000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:13.769 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.769+0000 7fc41effd640 3 client.4515.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:13.769 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.769+0000 7fc41effd640 3 client.4515.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:13.769 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.769+0000 7fc41effd640 20 client.4515.objecter dump_active .. 0 homeless 2024-03-25T19:48:13.769 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.769+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc41004f0a0 con 0x7fc4280ae280 2024-03-25T19:48:13.769 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.769+0000 7fc41effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:13.770 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:13.769+0000 7fc41effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:14.003 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.004+0000 7fc42cd82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:14.004 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.004+0000 7fc42cd82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:14.004 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.004+0000 7fc42cd82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:14.004 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.004+0000 7fc42cd82640 1 -- 172.21.15.81:0/1652813164 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc428146370 con 0x7fc4280ae280 2024-03-25T19:48:14.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.039+0000 7fc41effd640 1 -- 172.21.15.81:0/1652813164 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fc410050fd0 con 0x7fc4280ae280 2024-03-25T19:48:14.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.039+0000 7fc41effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:14.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.039+0000 7fc41effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:14.039 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:48:14.039 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:14.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.041+0000 7fc42cd82640 10 librados: watch_flush enter 2024-03-25T19:48:14.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.041+0000 7fc42cd82640 10 librados: watch_flush exit 2024-03-25T19:48:14.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.041+0000 7fc42cd82640 20 client.4515.objecter shutdown clearing up homeless session... 2024-03-25T19:48:14.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.041+0000 7fc42cd82640 10 client.4515.objecter successfully canceled tick 2024-03-25T19:48:14.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.041+0000 7fc42cd82640 1 -- 172.21.15.81:0/1652813164 >> v1:172.21.15.98:6800/3980829775 conn(0x7fc40806de50 legacy=0x7fc408070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:14.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.041+0000 7fc42cd82640 10 monclient: shutdown 2024-03-25T19:48:14.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.041+0000 7fc42cd82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:14.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.041+0000 7fc42cd82640 1 -- 172.21.15.81:0/1652813164 >> v1:172.21.15.81:6789/0 conn(0x7fc4280ae280 legacy=0x7fc4281da730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:14.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.041+0000 7fc42cd82640 1 -- 172.21.15.81:0/1652813164 shutdown_connections 2024-03-25T19:48:14.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.041+0000 7fc42cd82640 1 -- 172.21.15.81:0/1652813164 >> 172.21.15.81:0/1652813164 conn(0x7fc4280b0ef0 msgr2=0x7fc42813f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:14.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.042+0000 7fc42cd82640 1 -- 172.21.15.81:0/1652813164 shutdown_connections 2024-03-25T19:48:14.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.042+0000 7fc42cd82640 1 -- 172.21.15.81:0/1652813164 wait complete. 2024-03-25T19:48:14.043 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:14.042+0000 7fc42cd82640 1 librados: shutdown 2024-03-25T19:48:14.052 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:20.859 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:20.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.975+0000 7fdbab115640 10 monclient: get_monmap_and_config 2024-03-25T19:48:20.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.975+0000 7fdbab115640 10 monclient: build_initial_monmap 2024-03-25T19:48:20.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.976+0000 7fdbab115640 10 monclient: monmap: 2024-03-25T19:48:20.977 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:48:20.977 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:20.977 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:48:20.976896+0000 2024-03-25T19:48:20.978 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:48:20.976896+0000 2024-03-25T19:48:20.978 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:20.978 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:20.978 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:48:20.978 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:48:20.978 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:48:20.978 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:20.979 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.976+0000 7fdbab115640 1 Processor -- start 2024-03-25T19:48:20.979 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.977+0000 7fdbab115640 1 -- start start 2024-03-25T19:48:20.979 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.977+0000 7fdbab115640 10 monclient: init 2024-03-25T19:48:20.979 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.977+0000 7fdbab115640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:20.979 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.977+0000 7fdbab115640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:20.979 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.977+0000 7fdbab115640 10 monclient(hunting): picked mon.noname-c con 0x7fdba40a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:48:20.979 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.977+0000 7fdbab115640 10 monclient(hunting): picked mon.noname-b con 0x7fdba413f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:48:20.979 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.977+0000 7fdbab115640 10 monclient(hunting): picked mon.noname-a con 0x7fdba41430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:48:20.979 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.977+0000 7fdbab115640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba4142520 con 0x7fdba41430c0 2024-03-25T19:48:20.980 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.977+0000 7fdbab115640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba40b28a0 con 0x7fdba40a98f0 2024-03-25T19:48:20.980 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.977+0000 7fdbab115640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba40b2a30 con 0x7fdba413f710 2024-03-25T19:48:20.980 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.977+0000 7fdbab115640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:20.980 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.977+0000 7fdbab115640 10 monclient(hunting): authenticate will time out at 4336.059742s 2024-03-25T19:48:20.985 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.986+0000 7fdba3fff640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fdba413f710 0x7fdba41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:54412/0 (socket says 172.21.15.81:54412) 2024-03-25T19:48:20.985 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.986+0000 7fdba3fff640 1 -- 172.21.15.81:0/3852477271 learned_addr learned my addr 172.21.15.81:0/3852477271 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:48:20.992 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.992+0000 7fdba37fe640 1 -- 172.21.15.81:0/3852477271 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807364108 0 0) 0x7fdba40b2a30 con 0x7fdba413f710 2024-03-25T19:48:20.992 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.992+0000 7fdba37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:20.992 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.992+0000 7fdba37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:20.992 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.992+0000 7fdba37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:20.992 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.992+0000 7fdba37fe640 10 monclient(hunting): my global_id is 4493 2024-03-25T19:48:20.992 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:20.992+0000 7fdba37fe640 1 -- 172.21.15.81:0/3852477271 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb94003520 con 0x7fdba413f710 2024-03-25T19:48:21.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.014+0000 7fdba37fe640 1 -- 172.21.15.81:0/3852477271 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717523682 0 0) 0x7fdba4142520 con 0x7fdba41430c0 2024-03-25T19:48:21.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.014+0000 7fdba37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:21.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.014+0000 7fdba37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:21.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.014+0000 7fdba37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:21.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.014+0000 7fdba37fe640 10 monclient(hunting): my global_id is 4518 2024-03-25T19:48:21.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.014+0000 7fdba37fe640 1 -- 172.21.15.81:0/3852477271 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdba40b2a30 con 0x7fdba41430c0 2024-03-25T19:48:21.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.016+0000 7fdba37fe640 1 -- 172.21.15.81:0/3852477271 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2871295220 0 0) 0x7fdb94003520 con 0x7fdba413f710 2024-03-25T19:48:21.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.016+0000 7fdba37fe640 1 -- 172.21.15.81:0/3852477271 >> v1:172.21.15.98:6789/0 conn(0x7fdba40a98f0 legacy=0x7fdba40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:21.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.016+0000 7fdba37fe640 1 -- 172.21.15.81:0/3852477271 >> v1:172.21.15.81:6789/0 conn(0x7fdba41430c0 legacy=0x7fdba414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:21.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.016+0000 7fdba37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:48:21.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.016+0000 7fdba37fe640 1 monclient: found mon.noname-b 2024-03-25T19:48:21.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.016+0000 7fdba37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:21.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.016+0000 7fdba37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:48:21.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.016+0000 7fdba37fe640 1 -- 172.21.15.81:0/3852477271 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba40b44c0 con 0x7fdba413f710 2024-03-25T19:48:21.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.016+0000 7fdba37fe640 10 monclient: _finish_auth 0 2024-03-25T19:48:21.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.016+0000 7fdba37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:48:21.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.016+0000 7fdba37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:21.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.016+0000 7fdba37fe640 1 -- 172.21.15.81:0/3852477271 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fdb90001690 con 0x7fdba413f710 2024-03-25T19:48:21.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.016+0000 7fdba37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:21.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.016+0000 7fdba37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:21.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.016+0000 7fdba37fe640 10 monclient: dump: 2024-03-25T19:48:21.017 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:21.017 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:21.017 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:21.017 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:21.017 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:21.017 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:21.017 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:21.017 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:21.017 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:21.018 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:21.018 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.016+0000 7fdbab115640 5 monclient: authenticate success, global_id 4493 2024-03-25T19:48:21.018 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.016+0000 7fdbab115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:21.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.037+0000 7fdba37fe640 1 -- 172.21.15.81:0/3852477271 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb900017f0 con 0x7fdba413f710 2024-03-25T19:48:21.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.037+0000 7fdba37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:21.037 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.037+0000 7fdba37fe640 1 -- 172.21.15.81:0/3852477271 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fdb90001c00 con 0x7fdba413f710 2024-03-25T19:48:21.037 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.037+0000 7fdbab115640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:21.037 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.037+0000 7fdbab115640 4 set_mon_vals no callback set 2024-03-25T19:48:21.037 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.037+0000 7fdba37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:21.037 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.037+0000 7fdba37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:21.037 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.037+0000 7fdba37fe640 10 monclient: dump: 2024-03-25T19:48:21.037 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:21.037 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:21.037 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:21.037 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:21.037 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:21.037 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:21.037 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:21.037 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:21.038 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:21.038 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:21.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.038+0000 7fdbab115640 10 monclient: shutdown 2024-03-25T19:48:21.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.038+0000 7fdbab115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:21.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.038+0000 7fdbab115640 1 -- 172.21.15.81:0/3852477271 >> v1:172.21.15.81:6790/0 conn(0x7fdba413f710 legacy=0x7fdba41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:21.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.038+0000 7fdbab115640 1 -- 172.21.15.81:0/3852477271 shutdown_connections 2024-03-25T19:48:21.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.038+0000 7fdbab115640 1 -- 172.21.15.81:0/3852477271 >> 172.21.15.81:0/3852477271 conn(0x7fdba40b7ea0 msgr2=0x7fdba40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:21.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.038+0000 7fdbab115640 1 -- 172.21.15.81:0/3852477271 shutdown_connections 2024-03-25T19:48:21.040 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.038+0000 7fdbab115640 1 -- 172.21.15.81:0/3852477271 wait complete. 2024-03-25T19:48:21.040 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.039+0000 7fdbab115640 10 monclient: build_initial_monmap 2024-03-25T19:48:21.040 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.039+0000 7fdbab115640 10 monclient: monmap: 2024-03-25T19:48:21.040 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:48:21.040 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:21.040 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:48:21.040 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:48:21.040 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:21.040 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:48:21.040 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:48:21.040 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:48:21.040 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:48:21.040 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:21.040 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.039+0000 7fdbab115640 1 librados: starting msgr at 2024-03-25T19:48:21.040 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.039+0000 7fdbab115640 1 librados: starting objecter 2024-03-25T19:48:21.041 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.039+0000 7fdbab115640 1 Processor -- start 2024-03-25T19:48:21.041 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.039+0000 7fdbab115640 1 -- start start 2024-03-25T19:48:21.041 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.039+0000 7fdbab115640 1 librados: setting wanted keys 2024-03-25T19:48:21.041 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.039+0000 7fdbab115640 1 librados: calling monclient init 2024-03-25T19:48:21.041 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.039+0000 7fdbab115640 10 monclient: init 2024-03-25T19:48:21.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.040+0000 7fdbab115640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:21.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.040+0000 7fdbab115640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:21.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.040+0000 7fdbab115640 10 monclient(hunting): picked mon.noname-a con 0x7fdba40a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:48:21.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.040+0000 7fdbab115640 10 monclient(hunting): picked mon.noname-b con 0x7fdba413f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:48:21.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.040+0000 7fdbab115640 10 monclient(hunting): picked mon.noname-c con 0x7fdba41430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:48:21.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.040+0000 7fdbab115640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba413be70 con 0x7fdba40a98f0 2024-03-25T19:48:21.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.040+0000 7fdbab115640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba4131660 con 0x7fdba413f710 2024-03-25T19:48:21.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.040+0000 7fdbab115640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba41317f0 con 0x7fdba41430c0 2024-03-25T19:48:21.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.040+0000 7fdbab115640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:21.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.040+0000 7fdbab115640 10 monclient(hunting): authenticate will time out at 4336.122521s 2024-03-25T19:48:21.049 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.049+0000 7fdba8eb1640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fdba40a98f0 0x7fdba4130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:59660/0 (socket says 172.21.15.81:59660) 2024-03-25T19:48:21.049 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.049+0000 7fdba8eb1640 1 -- 172.21.15.81:0/2436260327 learned_addr learned my addr 172.21.15.81:0/2436260327 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:48:21.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.051+0000 7fdba17fa640 10 client.?.objecter ms_handle_connect 0x7fdba41430c0 2024-03-25T19:48:21.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.051+0000 7fdba17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:21.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.051+0000 7fdba17fa640 10 client.?.objecter ms_handle_connect 0x7fdba413f710 2024-03-25T19:48:21.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.051+0000 7fdba17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:21.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.052+0000 7fdba17fa640 10 client.?.objecter ms_handle_connect 0x7fdba40a98f0 2024-03-25T19:48:21.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.052+0000 7fdba17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:21.054 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.055+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049067993 0 0) 0x7fdba4131660 con 0x7fdba413f710 2024-03-25T19:48:21.054 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.055+0000 7fdba17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:48:21.054 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.055+0000 7fdba17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:21.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.055+0000 7fdba17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:21.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.055+0000 7fdba17fa640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:48:21.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.055+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb88004eb0 con 0x7fdba413f710 2024-03-25T19:48:21.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.060+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529223656 0 0) 0x7fdba41317f0 con 0x7fdba41430c0 2024-03-25T19:48:21.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.060+0000 7fdba17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:48:21.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.060+0000 7fdba17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:21.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.060+0000 7fdba17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:21.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.060+0000 7fdba17fa640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:48:21.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.060+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdba4131660 con 0x7fdba41430c0 2024-03-25T19:48:21.065 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.065+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117832475 0 0) 0x7fdba413be70 con 0x7fdba40a98f0 2024-03-25T19:48:21.065 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.065+0000 7fdba17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:48:21.065 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.065+0000 7fdba17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:21.065 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.065+0000 7fdba17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:21.065 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.065+0000 7fdba17fa640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:48:21.065 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.065+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdba41317f0 con 0x7fdba40a98f0 2024-03-25T19:48:21.068 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.068+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3441019801 0 0) 0x7fdb88004eb0 con 0x7fdba413f710 2024-03-25T19:48:21.068 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.068+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdba413be70 con 0x7fdba413f710 2024-03-25T19:48:21.068 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.068+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fdb90001b00 con 0x7fdba413f710 2024-03-25T19:48:21.068 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.068+0000 7fdba17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:21.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.068+0000 7fdba17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:21.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.068+0000 7fdba17fa640 10 monclient(hunting): dump: 2024-03-25T19:48:21.069 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:21.069 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:21.069 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:21.069 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:21.069 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:21.069 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:21.069 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:21.069 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:21.069 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:21.069 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:21.070 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.068+0000 7fdba17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:21.070 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.070+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2359151339 0 0) 0x7fdba4131660 con 0x7fdba41430c0 2024-03-25T19:48:21.070 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.070+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb88004eb0 con 0x7fdba41430c0 2024-03-25T19:48:21.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.070+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fdb9c001b50 con 0x7fdba41430c0 2024-03-25T19:48:21.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.070+0000 7fdba17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:21.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.070+0000 7fdba17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:21.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.070+0000 7fdba17fa640 10 monclient(hunting): dump: 2024-03-25T19:48:21.071 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:21.071 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:21.071 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:21.071 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:21.071 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:21.071 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:21.071 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:21.071 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:21.071 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:21.072 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:21.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.076+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2184939436 0 0) 0x7fdba41317f0 con 0x7fdba40a98f0 2024-03-25T19:48:21.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.076+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdba4131660 con 0x7fdba40a98f0 2024-03-25T19:48:21.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.076+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fdb980018f0 con 0x7fdba40a98f0 2024-03-25T19:48:21.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.076+0000 7fdba17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:21.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.076+0000 7fdba17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:21.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.076+0000 7fdba17fa640 10 monclient(hunting): dump: 2024-03-25T19:48:21.077 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:21.077 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:21.077 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:21.077 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:21.077 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:21.077 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:21.078 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:21.078 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:21.078 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:21.078 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:21.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1198937004 0 0) 0x7fdb88004eb0 con 0x7fdba41430c0 2024-03-25T19:48:21.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 >> v1:172.21.15.98:6789/0 conn(0x7fdba413f710 legacy=0x7fdba41d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:21.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 >> v1:172.21.15.81:6789/0 conn(0x7fdba40a98f0 legacy=0x7fdba4130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:21.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdba17fa640 10 monclient: _finish_hunting 0 2024-03-25T19:48:21.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdba17fa640 1 monclient: found mon.c 2024-03-25T19:48:21.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdba17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:21.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdba17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:48:21.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba4131980 con 0x7fdba41430c0 2024-03-25T19:48:21.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdba17fa640 10 monclient: _finish_auth 0 2024-03-25T19:48:21.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdba17fa640 10 monclient: _check_auth_tickets 2024-03-25T19:48:21.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdba17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:21.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdbab115640 5 monclient: authenticate success, global_id 4496 2024-03-25T19:48:21.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdbab115640 10 monclient: _renew_subs 2024-03-25T19:48:21.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdbab115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:48:21.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdbab115640 1 -- 172.21.15.81:0/2436260327 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba4135c60 con 0x7fdba41430c0 2024-03-25T19:48:21.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdbab115640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:21.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdbab115640 10 monclient: _renew_subs 2024-03-25T19:48:21.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdbab115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:48:21.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdbab115640 1 -- 172.21.15.81:0/2436260327 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba40b7770 con 0x7fdba41430c0 2024-03-25T19:48:21.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.079+0000 7fdbab115640 1 librados: init done 2024-03-25T19:48:21.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.080+0000 7fdbab115640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:21.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.080+0000 7fdbab115640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:21.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.080+0000 7fdbab115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:48:21.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.080+0000 7fdbab115640 1 -- 172.21.15.81:0/2436260327 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba4074cc0 con 0x7fdba41430c0 2024-03-25T19:48:21.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.385+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb9c002060 con 0x7fdba41430c0 2024-03-25T19:48:21.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.385+0000 7fdba17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:21.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.385+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fdb9c003b10 con 0x7fdba41430c0 2024-03-25T19:48:21.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.385+0000 7fdba17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:21.385 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.385+0000 7fdba17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:21.386 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.385+0000 7fdba17fa640 10 monclient: dump: 2024-03-25T19:48:21.386 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:21.386 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:21.386 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:21.386 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:21.386 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:21.386 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:21.386 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:21.386 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:21.386 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:21.386 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:21.386 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.385+0000 7fdba2ffd640 4 set_mon_vals no callback set 2024-03-25T19:48:21.387 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.386+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fdb9c019ae0 con 0x7fdba41430c0 2024-03-25T19:48:21.387 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.387+0000 7fdba8eb1640 10 monclient: get_auth_request con 0x7fdb8806e0b0 auth_method 0 2024-03-25T19:48:21.387 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.387+0000 7fdba17fa640 10 client.4496.objecter ms_handle_connect 0x7fdb8806e0b0 2024-03-25T19:48:21.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.393+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fdb9c0538b0 con 0x7fdba41430c0 2024-03-25T19:48:21.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.393+0000 7fdba17fa640 10 client.4496.objecter ms_dispatch 0x7fdba4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:21.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.393+0000 7fdba17fa640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:21.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.393+0000 7fdba17fa640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:21.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.393+0000 7fdba17fa640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-25T19:48:21.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.393+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb9c04f1f0 con 0x7fdba41430c0 2024-03-25T19:48:21.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.393+0000 7fdba17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:21.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.393+0000 7fdba17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:21.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.627+0000 7fdbab115640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:21.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.627+0000 7fdbab115640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:21.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.627+0000 7fdbab115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:48:21.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.627+0000 7fdbab115640 1 -- 172.21.15.81:0/2436260327 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdba41371d0 con 0x7fdba41430c0 2024-03-25T19:48:21.922 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.922+0000 7fdba17fa640 1 -- 172.21.15.81:0/2436260327 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fdb9c0512f0 con 0x7fdba41430c0 2024-03-25T19:48:21.922 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.922+0000 7fdba17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:21.922 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.922+0000 7fdba17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:21.923 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:48:21.923 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:21.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.924+0000 7fdbab115640 10 librados: watch_flush enter 2024-03-25T19:48:21.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.924+0000 7fdbab115640 10 librados: watch_flush exit 2024-03-25T19:48:21.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.924+0000 7fdbab115640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-25T19:48:21.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.924+0000 7fdbab115640 10 client.4496.objecter successfully canceled tick 2024-03-25T19:48:21.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.925+0000 7fdbab115640 1 -- 172.21.15.81:0/2436260327 >> v1:172.21.15.98:6800/3980829775 conn(0x7fdb8806e0b0 legacy=0x7fdb88070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:21.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.925+0000 7fdbab115640 10 monclient: shutdown 2024-03-25T19:48:21.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.925+0000 7fdbab115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:21.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.925+0000 7fdbab115640 1 -- 172.21.15.81:0/2436260327 >> v1:172.21.15.81:6790/0 conn(0x7fdba41430c0 legacy=0x7fdba41e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:21.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.925+0000 7fdbab115640 1 -- 172.21.15.81:0/2436260327 shutdown_connections 2024-03-25T19:48:21.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.925+0000 7fdbab115640 1 -- 172.21.15.81:0/2436260327 >> 172.21.15.81:0/2436260327 conn(0x7fdba40b7ea0 msgr2=0x7fdba4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:21.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.925+0000 7fdbab115640 1 -- 172.21.15.81:0/2436260327 shutdown_connections 2024-03-25T19:48:21.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.925+0000 7fdbab115640 1 -- 172.21.15.81:0/2436260327 wait complete. 2024-03-25T19:48:21.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:21.925+0000 7fdbab115640 1 librados: shutdown 2024-03-25T19:48:21.936 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:22.732 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:06:45.374 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:22.740 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:06:45.382 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:22.775 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:06:45.417 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:22.790 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:45.424 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:22.798 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:45.433 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:22.818 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:06:45.460 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:22.825 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:06:45.459 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:22.829 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:06:45.471 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:22.853 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:06:45.495 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:28.743 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:28.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.863+0000 7f3d235d0640 10 monclient: get_monmap_and_config 2024-03-25T19:48:28.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.863+0000 7f3d235d0640 10 monclient: build_initial_monmap 2024-03-25T19:48:28.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.863+0000 7f3d235d0640 10 monclient: monmap: 2024-03-25T19:48:28.863 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:48:28.863 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:28.863 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:48:28.863945+0000 2024-03-25T19:48:28.863 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:48:28.863945+0000 2024-03-25T19:48:28.863 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:28.863 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:28.863 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:48:28.863 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:48:28.863 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:48:28.863 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:28.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.863+0000 7f3d235d0640 1 Processor -- start 2024-03-25T19:48:28.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.863+0000 7f3d235d0640 1 -- start start 2024-03-25T19:48:28.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.863+0000 7f3d235d0640 10 monclient: init 2024-03-25T19:48:28.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.864+0000 7f3d235d0640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.864+0000 7f3d235d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.864+0000 7f3d235d0640 10 monclient(hunting): picked mon.noname-c con 0x7f3d1c0705b0 addr v1:172.21.15.98:6789/0 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.864+0000 7f3d235d0640 10 monclient(hunting): picked mon.noname-b con 0x7f3d1c1402d0 addr v1:172.21.15.81:6790/0 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.864+0000 7f3d235d0640 10 monclient(hunting): picked mon.noname-a con 0x7f3d1c149ae0 addr v1:172.21.15.81:6789/0 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.864+0000 7f3d235d0640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c0718c0 con 0x7f3d1c149ae0 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.864+0000 7f3d235d0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c0714c0 con 0x7f3d1c0705b0 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.864+0000 7f3d235d0640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c072b50 con 0x7f3d1c1402d0 2024-03-25T19:48:28.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.864+0000 7f3d235d0640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:28.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.864+0000 7f3d235d0640 10 monclient(hunting): authenticate will time out at 4343.946571s 2024-03-25T19:48:28.873 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.874+0000 7f3d2136c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3d1c0705b0 0x7f3d1c13d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:54526/0 (socket says 172.21.15.81:54526) 2024-03-25T19:48:28.873 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.874+0000 7f3d2136c640 1 -- 172.21.15.81:0/4268089836 learned_addr learned my addr 172.21.15.81:0/4268089836 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:48:28.889 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.889+0000 7f3d13fff640 1 -- 172.21.15.81:0/4268089836 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220243787 0 0) 0x7f3d1c0714c0 con 0x7f3d1c0705b0 2024-03-25T19:48:28.889 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.889+0000 7f3d13fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:28.889 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.889+0000 7f3d13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:28.890 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.889+0000 7f3d13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:28.890 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.889+0000 7f3d13fff640 10 monclient(hunting): my global_id is 4516 2024-03-25T19:48:28.890 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.889+0000 7f3d13fff640 1 -- 172.21.15.81:0/4268089836 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d08003520 con 0x7f3d1c0705b0 2024-03-25T19:48:28.891 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.890+0000 7f3d13fff640 1 -- 172.21.15.81:0/4268089836 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2491686306 0 0) 0x7f3d1c072b50 con 0x7f3d1c1402d0 2024-03-25T19:48:28.891 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.890+0000 7f3d13fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:28.891 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.890+0000 7f3d13fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:28.891 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.890+0000 7f3d13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:28.891 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.890+0000 7f3d13fff640 10 monclient(hunting): my global_id is 4499 2024-03-25T19:48:28.891 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.890+0000 7f3d13fff640 1 -- 172.21.15.81:0/4268089836 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d1c0714c0 con 0x7f3d1c1402d0 2024-03-25T19:48:28.901 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d13fff640 1 -- 172.21.15.81:0/4268089836 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2857907353 0 0) 0x7f3d1c0714c0 con 0x7f3d1c1402d0 2024-03-25T19:48:28.901 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d13fff640 1 -- 172.21.15.81:0/4268089836 >> v1:172.21.15.98:6789/0 conn(0x7f3d1c0705b0 legacy=0x7f3d1c13d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:28.901 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d13fff640 1 -- 172.21.15.81:0/4268089836 >> v1:172.21.15.81:6789/0 conn(0x7f3d1c149ae0 legacy=0x7f3d1c14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:28.901 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d13fff640 10 monclient: _finish_hunting 0 2024-03-25T19:48:28.901 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d13fff640 1 monclient: found mon.noname-b 2024-03-25T19:48:28.901 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:28.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d13fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:48:28.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d13fff640 1 -- 172.21.15.81:0/4268089836 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c13dd10 con 0x7f3d1c1402d0 2024-03-25T19:48:28.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d13fff640 10 monclient: _finish_auth 0 2024-03-25T19:48:28.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d13fff640 10 monclient: _check_auth_tickets 2024-03-25T19:48:28.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:28.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d13fff640 1 -- 172.21.15.81:0/4268089836 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3d0c001670 con 0x7f3d1c1402d0 2024-03-25T19:48:28.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d235d0640 5 monclient: authenticate success, global_id 4499 2024-03-25T19:48:28.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d235d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:28.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:28.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d13fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:28.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d13fff640 10 monclient: dump: 2024-03-25T19:48:28.902 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:28.903 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:28.903 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:28.903 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:28.903 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:28.903 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:28.903 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:28.903 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:28.903 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:28.904 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:28.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.901+0000 7f3d235d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:28.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.906+0000 7f3d13fff640 1 -- 172.21.15.81:0/4268089836 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d0c0017d0 con 0x7f3d1c1402d0 2024-03-25T19:48:28.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.907+0000 7f3d13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:28.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.907+0000 7f3d13fff640 1 -- 172.21.15.81:0/4268089836 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3d0c001ba0 con 0x7f3d1c1402d0 2024-03-25T19:48:28.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.907+0000 7f3d13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:28.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.907+0000 7f3d13fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:28.906 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.907+0000 7f3d13fff640 10 monclient: dump: 2024-03-25T19:48:28.906 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:28.907 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:28.907 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:28.907 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:28.907 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:28.907 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:28.907 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:28.907 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:28.907 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:28.907 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:28.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.907+0000 7f3d235d0640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:28.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.907+0000 7f3d235d0640 4 set_mon_vals no callback set 2024-03-25T19:48:28.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.907+0000 7f3d235d0640 10 monclient: shutdown 2024-03-25T19:48:28.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.907+0000 7f3d235d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:28.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.907+0000 7f3d235d0640 1 -- 172.21.15.81:0/4268089836 >> v1:172.21.15.81:6790/0 conn(0x7f3d1c1402d0 legacy=0x7f3d1c140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:28.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.908+0000 7f3d235d0640 1 -- 172.21.15.81:0/4268089836 shutdown_connections 2024-03-25T19:48:28.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.908+0000 7f3d235d0640 1 -- 172.21.15.81:0/4268089836 >> 172.21.15.81:0/4268089836 conn(0x7f3d1c0b5d10 msgr2=0x7f3d1c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:28.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.908+0000 7f3d235d0640 1 -- 172.21.15.81:0/4268089836 shutdown_connections 2024-03-25T19:48:28.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.908+0000 7f3d235d0640 1 -- 172.21.15.81:0/4268089836 wait complete. 2024-03-25T19:48:28.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.908+0000 7f3d235d0640 10 monclient: build_initial_monmap 2024-03-25T19:48:28.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.908+0000 7f3d235d0640 10 monclient: monmap: 2024-03-25T19:48:28.909 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:48:28.909 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:28.909 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:48:28.909 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:48:28.910 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:28.910 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:48:28.910 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:48:28.910 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:48:28.910 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:48:28.910 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:28.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.909+0000 7f3d235d0640 1 librados: starting msgr at 2024-03-25T19:48:28.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.909+0000 7f3d235d0640 1 librados: starting objecter 2024-03-25T19:48:28.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.909+0000 7f3d235d0640 1 Processor -- start 2024-03-25T19:48:28.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.909+0000 7f3d235d0640 1 -- start start 2024-03-25T19:48:28.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.909+0000 7f3d235d0640 1 librados: setting wanted keys 2024-03-25T19:48:28.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.909+0000 7f3d235d0640 1 librados: calling monclient init 2024-03-25T19:48:28.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.909+0000 7f3d235d0640 10 monclient: init 2024-03-25T19:48:28.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.909+0000 7f3d235d0640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:28.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.909+0000 7f3d235d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:28.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.909+0000 7f3d235d0640 10 monclient(hunting): picked mon.noname-b con 0x7f3d1c0705b0 addr v1:172.21.15.98:6789/0 2024-03-25T19:48:28.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.909+0000 7f3d235d0640 10 monclient(hunting): picked mon.noname-c con 0x7f3d1c1402d0 addr v1:172.21.15.81:6790/0 2024-03-25T19:48:28.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.910+0000 7f3d235d0640 10 monclient(hunting): picked mon.noname-a con 0x7f3d1c149ae0 addr v1:172.21.15.81:6789/0 2024-03-25T19:48:28.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.910+0000 7f3d235d0640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c06d6c0 con 0x7f3d1c149ae0 2024-03-25T19:48:28.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.910+0000 7f3d235d0640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c06bc90 con 0x7f3d1c0705b0 2024-03-25T19:48:28.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.910+0000 7f3d235d0640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c134950 con 0x7f3d1c1402d0 2024-03-25T19:48:28.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.910+0000 7f3d235d0640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:28.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.910+0000 7f3d235d0640 10 monclient(hunting): authenticate will time out at 4343.991924s 2024-03-25T19:48:28.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.917+0000 7f3d2136c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f3d1c0705b0 0x7f3d1c06c400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:54574/0 (socket says 172.21.15.81:54574) 2024-03-25T19:48:28.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.917+0000 7f3d2136c640 1 -- 172.21.15.81:0/1333292421 learned_addr learned my addr 172.21.15.81:0/1333292421 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:48:28.923 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.923+0000 7f3d11ffb640 10 client.?.objecter ms_handle_connect 0x7f3d1c0705b0 2024-03-25T19:48:28.923 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.923+0000 7f3d11ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:28.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.932+0000 7f3d11ffb640 10 client.?.objecter ms_handle_connect 0x7f3d1c1402d0 2024-03-25T19:48:28.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.932+0000 7f3d11ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:28.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.934+0000 7f3d11ffb640 1 -- 172.21.15.81:0/1333292421 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624704875 0 0) 0x7f3d1c06bc90 con 0x7f3d1c0705b0 2024-03-25T19:48:28.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.934+0000 7f3d11ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:28.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.934+0000 7f3d11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:28.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.934+0000 7f3d11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:28.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.934+0000 7f3d11ffb640 10 monclient(hunting): my global_id is 4519 2024-03-25T19:48:28.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.934+0000 7f3d11ffb640 1 -- 172.21.15.81:0/1333292421 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d00004eb0 con 0x7f3d1c0705b0 2024-03-25T19:48:28.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.937+0000 7f3d11ffb640 10 client.?.objecter ms_handle_connect 0x7f3d1c149ae0 2024-03-25T19:48:28.936 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.937+0000 7f3d11ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:28.943 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.944+0000 7f3d11ffb640 1 -- 172.21.15.81:0/1333292421 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3477967797 0 0) 0x7f3d00004eb0 con 0x7f3d1c0705b0 2024-03-25T19:48:28.944 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.944+0000 7f3d11ffb640 1 -- 172.21.15.81:0/1333292421 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d1c06bc90 con 0x7f3d1c0705b0 2024-03-25T19:48:28.944 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.944+0000 7f3d11ffb640 1 -- 172.21.15.81:0/1333292421 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3d18001990 con 0x7f3d1c0705b0 2024-03-25T19:48:28.944 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.944+0000 7f3d11ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:28.944 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.944+0000 7f3d11ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:28.944 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.944+0000 7f3d11ffb640 10 monclient(hunting): dump: 2024-03-25T19:48:28.944 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:28.944 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:28.944 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:28.944 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:28.944 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:28.944 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:28.944 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:28.944 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:28.944 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:28.945 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:28.945 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.944+0000 7f3d11ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:28.946 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.947+0000 7f3d11ffb640 1 -- 172.21.15.81:0/1333292421 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982693615 0 0) 0x7f3d1c134950 con 0x7f3d1c1402d0 2024-03-25T19:48:28.947 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.947+0000 7f3d11ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:48:28.947 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.947+0000 7f3d11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:28.947 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.947+0000 7f3d11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:28.947 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.947+0000 7f3d11ffb640 10 monclient(hunting): my global_id is 4502 2024-03-25T19:48:28.947 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.947+0000 7f3d11ffb640 1 -- 172.21.15.81:0/1333292421 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d00004eb0 con 0x7f3d1c1402d0 2024-03-25T19:48:28.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.951+0000 7f3d11ffb640 1 -- 172.21.15.81:0/1333292421 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1727717671 0 0) 0x7f3d1c06bc90 con 0x7f3d1c0705b0 2024-03-25T19:48:28.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.951+0000 7f3d11ffb640 1 -- 172.21.15.81:0/1333292421 >> v1:172.21.15.81:6790/0 conn(0x7f3d1c1402d0 legacy=0x7f3d1c133b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:28.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.951+0000 7f3d11ffb640 1 -- 172.21.15.81:0/1333292421 >> v1:172.21.15.81:6789/0 conn(0x7f3d1c149ae0 legacy=0x7f3d1c134240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:28.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.951+0000 7f3d11ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:48:28.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.951+0000 7f3d11ffb640 1 monclient: found mon.b 2024-03-25T19:48:28.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.951+0000 7f3d11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:28.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.952+0000 7f3d11ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:48:28.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.952+0000 7f3d11ffb640 1 -- 172.21.15.81:0/1333292421 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c06d8f0 con 0x7f3d1c0705b0 2024-03-25T19:48:28.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.952+0000 7f3d11ffb640 10 monclient: _finish_auth 0 2024-03-25T19:48:28.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.952+0000 7f3d11ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:48:28.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.952+0000 7f3d11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:28.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.952+0000 7f3d235d0640 5 monclient: authenticate success, global_id 4519 2024-03-25T19:48:28.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.952+0000 7f3d235d0640 10 monclient: _renew_subs 2024-03-25T19:48:28.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.952+0000 7f3d235d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:48:28.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.952+0000 7f3d235d0640 1 -- 172.21.15.81:0/1333292421 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d1c06bee0 con 0x7f3d1c0705b0 2024-03-25T19:48:28.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.952+0000 7f3d235d0640 10 client.4519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:28.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.952+0000 7f3d235d0640 10 monclient: _renew_subs 2024-03-25T19:48:28.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.952+0000 7f3d235d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:48:28.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.952+0000 7f3d235d0640 1 -- 172.21.15.81:0/1333292421 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1c06cbd0 con 0x7f3d1c0705b0 2024-03-25T19:48:28.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.952+0000 7f3d235d0640 1 librados: init done 2024-03-25T19:48:28.954 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.953+0000 7f3d235d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:28.954 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.953+0000 7f3d235d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:28.954 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.953+0000 7f3d235d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:48:28.954 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:28.953+0000 7f3d235d0640 1 -- 172.21.15.81:0/1333292421 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d1c0627b0 con 0x7f3d1c0705b0 2024-03-25T19:48:29.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.221+0000 7f3d11ffb640 1 -- 172.21.15.81:0/1333292421 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d180047e0 con 0x7f3d1c0705b0 2024-03-25T19:48:29.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.221+0000 7f3d11ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:29.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.221+0000 7f3d11ffb640 1 -- 172.21.15.81:0/1333292421 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3d18004bb0 con 0x7f3d1c0705b0 2024-03-25T19:48:29.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.221+0000 7f3d11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:29.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.221+0000 7f3d11ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:29.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.221+0000 7f3d11ffb640 10 monclient: dump: 2024-03-25T19:48:29.221 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:29.221 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:29.221 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:29.222 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:29.222 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:29.222 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:29.222 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:29.222 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:29.222 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:29.222 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:29.222 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.221+0000 7f3d137fe640 4 set_mon_vals no callback set 2024-03-25T19:48:29.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.222+0000 7f3d11ffb640 1 -- 172.21.15.81:0/1333292421 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f3d18004dd0 con 0x7f3d1c0705b0 2024-03-25T19:48:29.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.223+0000 7f3d20b6b640 10 monclient: get_auth_request con 0x7f3d0006dd90 auth_method 0 2024-03-25T19:48:29.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.223+0000 7f3d11ffb640 10 client.4519.objecter ms_handle_connect 0x7f3d0006dd90 2024-03-25T19:48:29.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.227+0000 7f3d11ffb640 1 -- 172.21.15.81:0/1333292421 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f3d18054870 con 0x7f3d1c0705b0 2024-03-25T19:48:29.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.227+0000 7f3d11ffb640 10 client.4519.objecter ms_dispatch 0x7f3d1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:29.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.227+0000 7f3d11ffb640 3 client.4519.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:29.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.227+0000 7f3d11ffb640 3 client.4519.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:29.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.227+0000 7f3d11ffb640 20 client.4519.objecter dump_active .. 0 homeless 2024-03-25T19:48:29.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.227+0000 7f3d11ffb640 1 -- 172.21.15.81:0/1333292421 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d18050380 con 0x7f3d1c0705b0 2024-03-25T19:48:29.228 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.227+0000 7f3d11ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:29.228 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.227+0000 7f3d11ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:29.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.479+0000 7f3d235d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:29.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.479+0000 7f3d235d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:29.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.479+0000 7f3d235d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:48:29.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.479+0000 7f3d235d0640 1 -- 172.21.15.81:0/1333292421 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d1c06c070 con 0x7f3d1c0705b0 2024-03-25T19:48:29.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.501+0000 7f3d11ffb640 1 -- 172.21.15.81:0/1333292421 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f3d180522b0 con 0x7f3d1c0705b0 2024-03-25T19:48:29.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.501+0000 7f3d11ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:29.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.501+0000 7f3d11ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:29.501 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:48:29.501 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:29.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.503+0000 7f3d235d0640 10 librados: watch_flush enter 2024-03-25T19:48:29.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.503+0000 7f3d235d0640 10 librados: watch_flush exit 2024-03-25T19:48:29.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.503+0000 7f3d235d0640 20 client.4519.objecter shutdown clearing up homeless session... 2024-03-25T19:48:29.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.503+0000 7f3d235d0640 10 client.4519.objecter successfully canceled tick 2024-03-25T19:48:29.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.503+0000 7f3d235d0640 1 -- 172.21.15.81:0/1333292421 >> v1:172.21.15.98:6800/3980829775 conn(0x7f3d0006dd90 legacy=0x7f3d00070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:29.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.503+0000 7f3d235d0640 10 monclient: shutdown 2024-03-25T19:48:29.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.503+0000 7f3d235d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:29.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.503+0000 7f3d235d0640 1 -- 172.21.15.81:0/1333292421 >> v1:172.21.15.98:6789/0 conn(0x7f3d1c0705b0 legacy=0x7f3d1c06c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:29.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.504+0000 7f3d235d0640 1 -- 172.21.15.81:0/1333292421 shutdown_connections 2024-03-25T19:48:29.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.504+0000 7f3d235d0640 1 -- 172.21.15.81:0/1333292421 >> 172.21.15.81:0/1333292421 conn(0x7f3d1c0b5d10 msgr2=0x7f3d1c0b4a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:29.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.504+0000 7f3d235d0640 1 -- 172.21.15.81:0/1333292421 shutdown_connections 2024-03-25T19:48:29.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.504+0000 7f3d235d0640 1 -- 172.21.15.81:0/1333292421 wait complete. 2024-03-25T19:48:29.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:29.504+0000 7f3d235d0640 1 librados: shutdown 2024-03-25T19:48:29.515 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:36.322 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:36.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.435+0000 7fe2c2a63640 10 monclient: get_monmap_and_config 2024-03-25T19:48:36.437 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.435+0000 7fe2c2a63640 10 monclient: build_initial_monmap 2024-03-25T19:48:36.437 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.435+0000 7fe2c2a63640 10 monclient: monmap: 2024-03-25T19:48:36.437 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:48:36.437 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:36.437 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:48:36.437790+0000 2024-03-25T19:48:36.437 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:48:36.437790+0000 2024-03-25T19:48:36.437 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:36.437 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:36.437 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:48:36.437 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:48:36.437 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:48:36.437 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:36.438 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.437+0000 7fe2c2a63640 1 Processor -- start 2024-03-25T19:48:36.438 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.437+0000 7fe2c2a63640 1 -- start start 2024-03-25T19:48:36.438 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.437+0000 7fe2c2a63640 10 monclient: init 2024-03-25T19:48:36.438 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.438+0000 7fe2c2a63640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:36.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.438+0000 7fe2c2a63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:36.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.438+0000 7fe2c2a63640 10 monclient(hunting): picked mon.noname-b con 0x7fe2bc0a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:48:36.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.438+0000 7fe2c2a63640 10 monclient(hunting): picked mon.noname-a con 0x7fe2bc13f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:48:36.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.438+0000 7fe2c2a63640 10 monclient(hunting): picked mon.noname-c con 0x7fe2bc1430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:48:36.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.438+0000 7fe2c2a63640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2bc142520 con 0x7fe2bc13f710 2024-03-25T19:48:36.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.438+0000 7fe2c2a63640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2bc0b28a0 con 0x7fe2bc1430c0 2024-03-25T19:48:36.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.438+0000 7fe2c2a63640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2bc0b2a30 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.438+0000 7fe2c2a63640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:36.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.438+0000 7fe2c2a63640 10 monclient(hunting): authenticate will time out at 4351.521086s 2024-03-25T19:48:36.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.440+0000 7fe2b3fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe2bc13f710 0x7fe2bc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:59760/0 (socket says 172.21.15.81:59760) 2024-03-25T19:48:36.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.440+0000 7fe2b3fff640 1 -- 172.21.15.81:0/2811055464 learned_addr learned my addr 172.21.15.81:0/2811055464 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:48:36.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.455+0000 7fe2bb7fe640 1 -- 172.21.15.81:0/2811055464 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4080428977 0 0) 0x7fe2bc142520 con 0x7fe2bc13f710 2024-03-25T19:48:36.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.455+0000 7fe2bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:36.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.455+0000 7fe2bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:36.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.455+0000 7fe2bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:36.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.455+0000 7fe2bb7fe640 10 monclient(hunting): my global_id is 4530 2024-03-25T19:48:36.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.455+0000 7fe2bb7fe640 1 -- 172.21.15.81:0/2811055464 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a8003520 con 0x7fe2bc13f710 2024-03-25T19:48:36.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.463+0000 7fe2bb7fe640 1 -- 172.21.15.81:0/2811055464 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733569794 0 0) 0x7fe2bc0b2a30 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.463+0000 7fe2bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:36.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.463+0000 7fe2bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:36.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.463+0000 7fe2bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:36.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.463+0000 7fe2bb7fe640 10 monclient(hunting): my global_id is 4505 2024-03-25T19:48:36.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.463+0000 7fe2bb7fe640 1 -- 172.21.15.81:0/2811055464 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2bc142520 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2bb7fe640 1 -- 172.21.15.81:0/2811055464 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2187907313 0 0) 0x7fe2a8003520 con 0x7fe2bc13f710 2024-03-25T19:48:36.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2bb7fe640 1 -- 172.21.15.81:0/2811055464 >> v1:172.21.15.81:6790/0 conn(0x7fe2bc0a98f0 legacy=0x7fe2bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:36.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2bb7fe640 1 -- 172.21.15.81:0/2811055464 >> v1:172.21.15.98:6789/0 conn(0x7fe2bc1430c0 legacy=0x7fe2bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:36.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2bb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:48:36.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2bb7fe640 1 monclient: found mon.noname-a 2024-03-25T19:48:36.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:36.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2bb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:48:36.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2bb7fe640 1 -- 172.21.15.81:0/2811055464 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc0b44c0 con 0x7fe2bc13f710 2024-03-25T19:48:36.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2bb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:48:36.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2bb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:48:36.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:36.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2bb7fe640 1 -- 172.21.15.81:0/2811055464 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe2a4001690 con 0x7fe2bc13f710 2024-03-25T19:48:36.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2c2a63640 5 monclient: authenticate success, global_id 4530 2024-03-25T19:48:36.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2c2a63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:36.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:36.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2bb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:36.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2bb7fe640 10 monclient: dump: 2024-03-25T19:48:36.478 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:36.478 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:36.478 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:36.478 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:36.478 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:36.478 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:36.478 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:36.478 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:36.478 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:36.478 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:36.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.475+0000 7fe2c2a63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:36.484 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.484+0000 7fe2bb7fe640 1 -- 172.21.15.81:0/2811055464 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2a40017f0 con 0x7fe2bc13f710 2024-03-25T19:48:36.484 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.484+0000 7fe2bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:36.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.484+0000 7fe2bb7fe640 1 -- 172.21.15.81:0/2811055464 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe2a4001c00 con 0x7fe2bc13f710 2024-03-25T19:48:36.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.484+0000 7fe2c2a63640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:36.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.484+0000 7fe2c2a63640 4 set_mon_vals no callback set 2024-03-25T19:48:36.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.484+0000 7fe2bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:36.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.484+0000 7fe2bb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:36.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.484+0000 7fe2bb7fe640 10 monclient: dump: 2024-03-25T19:48:36.485 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:36.485 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:36.485 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:36.485 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:36.485 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:36.485 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:36.485 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:36.485 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:36.486 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:36.486 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:36.487 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.485+0000 7fe2c2a63640 10 monclient: shutdown 2024-03-25T19:48:36.487 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.485+0000 7fe2c2a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:36.487 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.485+0000 7fe2c2a63640 1 -- 172.21.15.81:0/2811055464 >> v1:172.21.15.81:6789/0 conn(0x7fe2bc13f710 legacy=0x7fe2bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:36.487 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.485+0000 7fe2c2a63640 1 -- 172.21.15.81:0/2811055464 shutdown_connections 2024-03-25T19:48:36.487 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.485+0000 7fe2c2a63640 1 -- 172.21.15.81:0/2811055464 >> 172.21.15.81:0/2811055464 conn(0x7fe2bc0b7ea0 msgr2=0x7fe2bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:36.487 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.485+0000 7fe2c2a63640 1 -- 172.21.15.81:0/2811055464 shutdown_connections 2024-03-25T19:48:36.487 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.485+0000 7fe2c2a63640 1 -- 172.21.15.81:0/2811055464 wait complete. 2024-03-25T19:48:36.487 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.486+0000 7fe2c2a63640 10 monclient: build_initial_monmap 2024-03-25T19:48:36.487 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.486+0000 7fe2c2a63640 10 monclient: monmap: 2024-03-25T19:48:36.487 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:48:36.487 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:36.488 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:48:36.488 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:48:36.488 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:36.488 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:48:36.488 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:48:36.488 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:48:36.488 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:48:36.488 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:36.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.486+0000 7fe2c2a63640 1 librados: starting msgr at 2024-03-25T19:48:36.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.486+0000 7fe2c2a63640 1 librados: starting objecter 2024-03-25T19:48:36.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.486+0000 7fe2c2a63640 1 Processor -- start 2024-03-25T19:48:36.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.486+0000 7fe2c2a63640 1 -- start start 2024-03-25T19:48:36.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.486+0000 7fe2c2a63640 1 librados: setting wanted keys 2024-03-25T19:48:36.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.486+0000 7fe2c2a63640 1 librados: calling monclient init 2024-03-25T19:48:36.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.486+0000 7fe2c2a63640 10 monclient: init 2024-03-25T19:48:36.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.487+0000 7fe2c2a63640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:36.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.487+0000 7fe2c2a63640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:36.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.487+0000 7fe2c2a63640 10 monclient(hunting): picked mon.noname-a con 0x7fe2bc0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:48:36.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.487+0000 7fe2c2a63640 10 monclient(hunting): picked mon.noname-c con 0x7fe2bc13f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:48:36.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.487+0000 7fe2c2a63640 10 monclient(hunting): picked mon.noname-b con 0x7fe2bc1430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:48:36.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.487+0000 7fe2c2a63640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2bc1d82d0 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.487+0000 7fe2c2a63640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2bc0abb90 con 0x7fe2bc1430c0 2024-03-25T19:48:36.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.487+0000 7fe2c2a63640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2bc1d76a0 con 0x7fe2bc13f710 2024-03-25T19:48:36.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.488+0000 7fe2c2a63640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:36.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.488+0000 7fe2c2a63640 10 monclient(hunting): authenticate will time out at 4351.570834s 2024-03-25T19:48:36.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.503+0000 7fe2b3fff640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fe2bc13f710 0x7fe2bc1d7bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:41008/0 (socket says 172.21.15.81:41008) 2024-03-25T19:48:36.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.503+0000 7fe2b3fff640 1 -- 172.21.15.81:0/11497886 learned_addr learned my addr 172.21.15.81:0/11497886 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:48:36.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.504+0000 7fe2b97fa640 10 client.?.objecter ms_handle_connect 0x7fe2bc13f710 2024-03-25T19:48:36.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.504+0000 7fe2b97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:36.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.508+0000 7fe2b97fa640 10 client.?.objecter ms_handle_connect 0x7fe2bc0a98f0 2024-03-25T19:48:36.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.508+0000 7fe2b97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:36.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.513+0000 7fe2b97fa640 10 client.?.objecter ms_handle_connect 0x7fe2bc1430c0 2024-03-25T19:48:36.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.513+0000 7fe2b97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:36.520 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.519+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890605177 0 0) 0x7fe2bc1d76a0 con 0x7fe2bc13f710 2024-03-25T19:48:36.520 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.519+0000 7fe2b97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:48:36.520 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.519+0000 7fe2b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:36.520 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.519+0000 7fe2b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:36.520 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.519+0000 7fe2b97fa640 10 monclient(hunting): my global_id is 4508 2024-03-25T19:48:36.520 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.519+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe29c004eb0 con 0x7fe2bc13f710 2024-03-25T19:48:36.525 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.525+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220890068 0 0) 0x7fe2bc1d82d0 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.526 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.525+0000 7fe2b97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:48:36.526 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.525+0000 7fe2b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:36.526 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.525+0000 7fe2b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:36.526 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.525+0000 7fe2b97fa640 10 monclient(hunting): my global_id is 4533 2024-03-25T19:48:36.526 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.525+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2bc1d76a0 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.526 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.526+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633138669 0 0) 0x7fe2bc0abb90 con 0x7fe2bc1430c0 2024-03-25T19:48:36.526 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.526+0000 7fe2b97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:48:36.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.526+0000 7fe2b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:36.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.526+0000 7fe2b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:36.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.526+0000 7fe2b97fa640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:48:36.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.526+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2bc1d82d0 con 0x7fe2bc1430c0 2024-03-25T19:48:36.537 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.536+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1571468902 0 0) 0x7fe2bc1d82d0 con 0x7fe2bc1430c0 2024-03-25T19:48:36.537 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.536+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2bc0abb90 con 0x7fe2bc1430c0 2024-03-25T19:48:36.537 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.536+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe2b4001a90 con 0x7fe2bc1430c0 2024-03-25T19:48:36.537 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.536+0000 7fe2b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:36.537 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.536+0000 7fe2b97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:36.537 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.536+0000 7fe2b97fa640 10 monclient(hunting): dump: 2024-03-25T19:48:36.537 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:36.537 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.536+0000 7fe2b97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:36.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.539+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3294749980 0 0) 0x7fe29c004eb0 con 0x7fe2bc13f710 2024-03-25T19:48:36.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.539+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2bc1d82d0 con 0x7fe2bc13f710 2024-03-25T19:48:36.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.539+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe2a4001bc0 con 0x7fe2bc13f710 2024-03-25T19:48:36.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.539+0000 7fe2b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:36.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.539+0000 7fe2b97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:36.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.539+0000 7fe2b97fa640 10 monclient(hunting): dump: 2024-03-25T19:48:36.540 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:36.540 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:36.540 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:36.540 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:36.540 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:36.540 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:36.540 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:36.541 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:36.541 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:36.541 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:36.541 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.540+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 202436034 0 0) 0x7fe2bc1d76a0 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.541 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.540+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe29c004eb0 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.541 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.540+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe2ac001930 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.542 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.540+0000 7fe2b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:36.542 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.540+0000 7fe2b97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:36.542 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.540+0000 7fe2b97fa640 10 monclient(hunting): dump: 2024-03-25T19:48:36.542 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:36.542 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:36.542 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:36.542 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:36.542 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:36.542 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:36.542 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:36.542 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:36.545 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:36.545 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:36.547 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3014162021 0 0) 0x7fe29c004eb0 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.547 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 >> v1:172.21.15.81:6790/0 conn(0x7fe2bc13f710 legacy=0x7fe2bc1d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:36.547 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 >> v1:172.21.15.98:6789/0 conn(0x7fe2bc1430c0 legacy=0x7fe2bc134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:36.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2b97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:48:36.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2b97fa640 1 monclient: found mon.a 2024-03-25T19:48:36.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:36.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2b97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:36.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc134720 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2b97fa640 10 monclient: _finish_auth 0 2024-03-25T19:48:36.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2b97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:48:36.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:36.548 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2c2a63640 5 monclient: authenticate success, global_id 4533 2024-03-25T19:48:36.549 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2c2a63640 10 monclient: _renew_subs 2024-03-25T19:48:36.549 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2c2a63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:36.549 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2c2a63640 1 -- 172.21.15.81:0/11497886 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2bc134970 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.549 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2c2a63640 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:36.549 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2c2a63640 10 monclient: _renew_subs 2024-03-25T19:48:36.549 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2c2a63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:36.549 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2c2a63640 1 -- 172.21.15.81:0/11497886 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2bc137a10 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.549 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.546+0000 7fe2c2a63640 1 librados: init done 2024-03-25T19:48:36.549 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.547+0000 7fe2c2a63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:36.549 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.547+0000 7fe2c2a63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:36.549 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.547+0000 7fe2c2a63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:36.549 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.547+0000 7fe2c2a63640 1 -- 172.21.15.81:0/11497886 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2bc074cc0 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.841 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.840+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2ac003610 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.841 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.840+0000 7fe2b97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:36.841 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.840+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe2ac003920 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.841 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.840+0000 7fe2b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:36.841 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.840+0000 7fe2b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:36.841 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.840+0000 7fe2b97fa640 10 monclient: dump: 2024-03-25T19:48:36.841 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:36.841 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:36.841 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:36.842 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:36.842 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:36.842 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:36.842 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:36.842 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:36.842 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:36.842 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:36.842 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.840+0000 7fe2baffd640 4 set_mon_vals no callback set 2024-03-25T19:48:36.843 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.841+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fe2ac0198f0 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.843 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.842+0000 7fe2b3fff640 10 monclient: get_auth_request con 0x7fe29c06e160 auth_method 0 2024-03-25T19:48:36.843 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.842+0000 7fe2b97fa640 10 client.4533.objecter ms_handle_connect 0x7fe29c06e160 2024-03-25T19:48:36.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.846+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fe2ac053560 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.846+0000 7fe2b97fa640 10 client.4533.objecter ms_dispatch 0x7fe2bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:36.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.846+0000 7fe2b97fa640 3 client.4533.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:36.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.846+0000 7fe2b97fa640 3 client.4533.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:36.848 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.846+0000 7fe2b97fa640 20 client.4533.objecter dump_active .. 0 homeless 2024-03-25T19:48:36.848 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.846+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe2ac04f070 con 0x7fe2bc0a98f0 2024-03-25T19:48:36.848 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.846+0000 7fe2b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:36.848 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:36.846+0000 7fe2b97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:37.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.088+0000 7fe2c2a63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:37.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.088+0000 7fe2c2a63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:37.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.088+0000 7fe2c2a63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:37.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.088+0000 7fe2c2a63640 1 -- 172.21.15.81:0/11497886 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2bc1d7830 con 0x7fe2bc0a98f0 2024-03-25T19:48:37.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.112+0000 7fe2b97fa640 1 -- 172.21.15.81:0/11497886 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fe2ac050fa0 con 0x7fe2bc0a98f0 2024-03-25T19:48:37.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.112+0000 7fe2b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:37.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.112+0000 7fe2b97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:37.113 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:48:37.113 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:37.116 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.115+0000 7fe2c2a63640 10 librados: watch_flush enter 2024-03-25T19:48:37.116 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.115+0000 7fe2c2a63640 10 librados: watch_flush exit 2024-03-25T19:48:37.116 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.115+0000 7fe2c2a63640 20 client.4533.objecter shutdown clearing up homeless session... 2024-03-25T19:48:37.117 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.115+0000 7fe2c2a63640 10 client.4533.objecter successfully canceled tick 2024-03-25T19:48:37.117 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.115+0000 7fe2c2a63640 1 -- 172.21.15.81:0/11497886 >> v1:172.21.15.98:6800/3980829775 conn(0x7fe29c06e160 legacy=0x7fe29c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:37.117 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.115+0000 7fe2c2a63640 10 monclient: shutdown 2024-03-25T19:48:37.117 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.115+0000 7fe2c2a63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:37.117 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.115+0000 7fe2c2a63640 1 -- 172.21.15.81:0/11497886 >> v1:172.21.15.81:6789/0 conn(0x7fe2bc0a98f0 legacy=0x7fe2bc1385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:37.117 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.115+0000 7fe2c2a63640 1 -- 172.21.15.81:0/11497886 shutdown_connections 2024-03-25T19:48:37.117 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.115+0000 7fe2c2a63640 1 -- 172.21.15.81:0/11497886 >> 172.21.15.81:0/11497886 conn(0x7fe2bc0b7ea0 msgr2=0x7fe2bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:37.117 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.115+0000 7fe2c2a63640 1 -- 172.21.15.81:0/11497886 shutdown_connections 2024-03-25T19:48:37.117 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.115+0000 7fe2c2a63640 1 -- 172.21.15.81:0/11497886 wait complete. 2024-03-25T19:48:37.117 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:37.115+0000 7fe2c2a63640 1 librados: shutdown 2024-03-25T19:48:37.128 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:42.760 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:07:05.401 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:42.805 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:07:05.447 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:42.836 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:07:05.477 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:42.872 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:07:05.514 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:42.876 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:07:05.518 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:42.888 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:05.523 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:42.893 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:05.528 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:42.903 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:07:05.545 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:42.926 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:05.560 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:43.935 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:44.032 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.032+0000 7fed35c25640 10 monclient: get_monmap_and_config 2024-03-25T19:48:44.032 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.032+0000 7fed35c25640 10 monclient: build_initial_monmap 2024-03-25T19:48:44.032 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.032+0000 7fed35c25640 10 monclient: monmap: 2024-03-25T19:48:44.032 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:48:44.032 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:44.032 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:48:44.033276+0000 2024-03-25T19:48:44.032 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:48:44.033276+0000 2024-03-25T19:48:44.032 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:44.032 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:44.032 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:48:44.032 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:48:44.033 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:48:44.033 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:44.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.033+0000 7fed35c25640 1 Processor -- start 2024-03-25T19:48:44.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.033+0000 7fed35c25640 1 -- start start 2024-03-25T19:48:44.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.033+0000 7fed35c25640 10 monclient: init 2024-03-25T19:48:44.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.033+0000 7fed35c25640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:44.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.033+0000 7fed35c25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:44.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.033+0000 7fed35c25640 10 monclient(hunting): picked mon.noname-c con 0x7fed300b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:48:44.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.033+0000 7fed35c25640 10 monclient(hunting): picked mon.noname-b con 0x7fed300b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:48:44.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.033+0000 7fed35c25640 10 monclient(hunting): picked mon.noname-a con 0x7fed3014be90 addr v1:172.21.15.81:6789/0 2024-03-25T19:48:44.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.033+0000 7fed35c25640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed300af940 con 0x7fed3014be90 2024-03-25T19:48:44.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.033+0000 7fed35c25640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed300b0a70 con 0x7fed300b4d60 2024-03-25T19:48:44.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.033+0000 7fed35c25640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed300b0d50 con 0x7fed300b1730 2024-03-25T19:48:44.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.033+0000 7fed35c25640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:44.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.033+0000 7fed35c25640 10 monclient(hunting): authenticate will time out at 4359.115679s 2024-03-25T19:48:44.054 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.055+0000 7fed2ffff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fed3014be90 0x7fed3014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:50782/0 (socket says 172.21.15.81:50782) 2024-03-25T19:48:44.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.055+0000 7fed2ffff640 1 -- 172.21.15.81:0/2411758260 learned_addr learned my addr 172.21.15.81:0/2411758260 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:48:44.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.069+0000 7fed2e7fc640 1 -- 172.21.15.81:0/2411758260 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215352931 0 0) 0x7fed300af940 con 0x7fed3014be90 2024-03-25T19:48:44.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.069+0000 7fed2e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:44.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.069+0000 7fed2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:44.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.069+0000 7fed2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:44.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.069+0000 7fed2e7fc640 10 monclient(hunting): my global_id is 4536 2024-03-25T19:48:44.069 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.069+0000 7fed2e7fc640 1 -- 172.21.15.81:0/2411758260 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed18003520 con 0x7fed3014be90 2024-03-25T19:48:44.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed2e7fc640 1 -- 172.21.15.81:0/2411758260 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2139064057 0 0) 0x7fed18003520 con 0x7fed3014be90 2024-03-25T19:48:44.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed2e7fc640 1 -- 172.21.15.81:0/2411758260 >> v1:172.21.15.81:6790/0 conn(0x7fed300b1730 legacy=0x7fed300b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:44.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed2e7fc640 1 -- 172.21.15.81:0/2411758260 >> v1:172.21.15.98:6789/0 conn(0x7fed300b4d60 legacy=0x7fed3013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:44.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed2e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:48:44.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed2e7fc640 1 monclient: found mon.noname-a 2024-03-25T19:48:44.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:44.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed2e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:48:44.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed2e7fc640 1 -- 172.21.15.81:0/2411758260 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed300b1030 con 0x7fed3014be90 2024-03-25T19:48:44.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed2e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:48:44.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed2e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:48:44.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:44.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed2e7fc640 1 -- 172.21.15.81:0/2411758260 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fed28001a10 con 0x7fed3014be90 2024-03-25T19:48:44.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed35c25640 5 monclient: authenticate success, global_id 4536 2024-03-25T19:48:44.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed35c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:44.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:44.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed2e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:44.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed2e7fc640 10 monclient: dump: 2024-03-25T19:48:44.083 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:44.083 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:44.083 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:44.083 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:44.083 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:44.083 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:44.083 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:44.083 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:44.083 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:44.084 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:44.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.082+0000 7fed35c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:44.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.086+0000 7fed2e7fc640 1 -- 172.21.15.81:0/2411758260 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed28001b70 con 0x7fed3014be90 2024-03-25T19:48:44.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.086+0000 7fed2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:44.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.086+0000 7fed2e7fc640 1 -- 172.21.15.81:0/2411758260 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fed28001f80 con 0x7fed3014be90 2024-03-25T19:48:44.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.086+0000 7fed2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:44.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.086+0000 7fed2e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:44.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.086+0000 7fed2e7fc640 10 monclient: dump: 2024-03-25T19:48:44.086 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:44.086 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:44.086 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:44.086 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:44.086 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:44.086 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:44.087 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:44.087 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:44.087 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:44.087 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:44.087 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.086+0000 7fed35c25640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:44.087 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.086+0000 7fed35c25640 4 set_mon_vals no callback set 2024-03-25T19:48:44.088 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.087+0000 7fed35c25640 10 monclient: shutdown 2024-03-25T19:48:44.088 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.087+0000 7fed35c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:44.088 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.087+0000 7fed35c25640 1 -- 172.21.15.81:0/2411758260 >> v1:172.21.15.81:6789/0 conn(0x7fed3014be90 legacy=0x7fed3014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:44.088 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.087+0000 7fed35c25640 1 -- 172.21.15.81:0/2411758260 shutdown_connections 2024-03-25T19:48:44.088 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.087+0000 7fed35c25640 1 -- 172.21.15.81:0/2411758260 >> 172.21.15.81:0/2411758260 conn(0x7fed300617a0 msgr2=0x7fed30061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:44.088 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.087+0000 7fed35c25640 1 -- 172.21.15.81:0/2411758260 shutdown_connections 2024-03-25T19:48:44.088 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.087+0000 7fed35c25640 1 -- 172.21.15.81:0/2411758260 wait complete. 2024-03-25T19:48:44.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.088+0000 7fed35c25640 10 monclient: build_initial_monmap 2024-03-25T19:48:44.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.088+0000 7fed35c25640 10 monclient: monmap: 2024-03-25T19:48:44.089 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:48:44.089 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:44.089 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:48:44.089 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:48:44.089 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:44.089 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:48:44.089 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:48:44.089 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:48:44.089 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:48:44.089 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:44.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.088+0000 7fed35c25640 1 librados: starting msgr at 2024-03-25T19:48:44.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.088+0000 7fed35c25640 1 librados: starting objecter 2024-03-25T19:48:44.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.089+0000 7fed35c25640 1 Processor -- start 2024-03-25T19:48:44.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.089+0000 7fed35c25640 1 -- start start 2024-03-25T19:48:44.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.089+0000 7fed35c25640 1 librados: setting wanted keys 2024-03-25T19:48:44.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.089+0000 7fed35c25640 1 librados: calling monclient init 2024-03-25T19:48:44.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.089+0000 7fed35c25640 10 monclient: init 2024-03-25T19:48:44.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.090+0000 7fed35c25640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:44.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.090+0000 7fed35c25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:44.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.090+0000 7fed35c25640 10 monclient(hunting): picked mon.noname-a con 0x7fed300b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:48:44.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.090+0000 7fed35c25640 10 monclient(hunting): picked mon.noname-b con 0x7fed300b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:48:44.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.090+0000 7fed35c25640 10 monclient(hunting): picked mon.noname-c con 0x7fed3014be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:48:44.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.090+0000 7fed35c25640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed301307c0 con 0x7fed300b1730 2024-03-25T19:48:44.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.090+0000 7fed35c25640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed30138f90 con 0x7fed300b4d60 2024-03-25T19:48:44.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.090+0000 7fed35c25640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed30132ab0 con 0x7fed3014be90 2024-03-25T19:48:44.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.090+0000 7fed35c25640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:44.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.090+0000 7fed35c25640 10 monclient(hunting): authenticate will time out at 4359.172313s 2024-03-25T19:48:44.102 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.103+0000 7fed2ffff640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fed3014be90 0x7fed301300b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:47984/0 (socket says 172.21.15.81:47984) 2024-03-25T19:48:44.102 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.103+0000 7fed2ffff640 1 -- 172.21.15.81:0/340392850 learned_addr learned my addr 172.21.15.81:0/340392850 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:48:44.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.104+0000 7fed17fff640 10 client.?.objecter ms_handle_connect 0x7fed3014be90 2024-03-25T19:48:44.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.104+0000 7fed17fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:44.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.105+0000 7fed17fff640 10 client.?.objecter ms_handle_connect 0x7fed300b1730 2024-03-25T19:48:44.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.105+0000 7fed17fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:44.109 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.110+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978394932 0 0) 0x7fed301307c0 con 0x7fed300b1730 2024-03-25T19:48:44.109 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.110+0000 7fed17fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:44.109 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.110+0000 7fed17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:44.110 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.110+0000 7fed17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:44.110 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.110+0000 7fed17fff640 10 monclient(hunting): my global_id is 4539 2024-03-25T19:48:44.110 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.110+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed1c004eb0 con 0x7fed300b1730 2024-03-25T19:48:44.115 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.116+0000 7fed17fff640 10 client.?.objecter ms_handle_connect 0x7fed300b4d60 2024-03-25T19:48:44.115 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.116+0000 7fed17fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:44.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.129+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1436799177 0 0) 0x7fed30132ab0 con 0x7fed3014be90 2024-03-25T19:48:44.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.129+0000 7fed17fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:44.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.129+0000 7fed17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:44.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.129+0000 7fed17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:44.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.129+0000 7fed17fff640 10 monclient(hunting): my global_id is 4514 2024-03-25T19:48:44.129 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.129+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed301307c0 con 0x7fed3014be90 2024-03-25T19:48:44.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.132+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 113086247 0 0) 0x7fed1c004eb0 con 0x7fed300b1730 2024-03-25T19:48:44.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.132+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed30132ab0 con 0x7fed300b1730 2024-03-25T19:48:44.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.132+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fed20001930 con 0x7fed300b1730 2024-03-25T19:48:44.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.132+0000 7fed17fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:44.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.132+0000 7fed17fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:44.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.132+0000 7fed17fff640 10 monclient(hunting): dump: 2024-03-25T19:48:44.132 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:44.132 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:44.132 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:44.132 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:44.132 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:44.132 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:44.133 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:44.133 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:44.133 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:44.133 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:44.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.132+0000 7fed17fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:44.138 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.138+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474443704 0 0) 0x7fed30138f90 con 0x7fed300b4d60 2024-03-25T19:48:44.138 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.138+0000 7fed17fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:44.138 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.138+0000 7fed17fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:44.138 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.138+0000 7fed17fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:44.138 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.138+0000 7fed17fff640 10 monclient(hunting): my global_id is 4531 2024-03-25T19:48:44.138 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.138+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed1c004eb0 con 0x7fed300b4d60 2024-03-25T19:48:44.146 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.146+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2740274303 0 0) 0x7fed301307c0 con 0x7fed3014be90 2024-03-25T19:48:44.146 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.146+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed30138f90 con 0x7fed3014be90 2024-03-25T19:48:44.146 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.146+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fed28001e80 con 0x7fed3014be90 2024-03-25T19:48:44.146 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.146+0000 7fed17fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:44.146 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.146+0000 7fed17fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:44.146 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.146+0000 7fed17fff640 10 monclient(hunting): dump: 2024-03-25T19:48:44.146 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:44.146 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:44.146 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:44.147 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:44.147 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:44.147 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:44.147 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:44.147 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:44.147 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:44.147 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:44.151 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.151+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 393959409 0 0) 0x7fed1c004eb0 con 0x7fed300b4d60 2024-03-25T19:48:44.151 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.151+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed301307c0 con 0x7fed300b4d60 2024-03-25T19:48:44.151 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.151+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fed24001830 con 0x7fed300b4d60 2024-03-25T19:48:44.151 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.151+0000 7fed17fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:44.151 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.151+0000 7fed17fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:44.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.151+0000 7fed17fff640 10 monclient(hunting): dump: 2024-03-25T19:48:44.152 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:44.152 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:44.152 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:44.152 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:44.152 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:44.152 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:44.152 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:44.152 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:44.152 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:44.152 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:44.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3782926309 0 0) 0x7fed30132ab0 con 0x7fed300b1730 2024-03-25T19:48:44.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 >> v1:172.21.15.81:6790/0 conn(0x7fed3014be90 legacy=0x7fed301300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:44.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 >> v1:172.21.15.98:6789/0 conn(0x7fed300b4d60 legacy=0x7fed3012f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:44.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed17fff640 10 monclient: _finish_hunting 0 2024-03-25T19:48:44.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed17fff640 1 monclient: found mon.a 2024-03-25T19:48:44.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:44.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed17fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:44.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed30133a40 con 0x7fed300b1730 2024-03-25T19:48:44.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed17fff640 10 monclient: _finish_auth 0 2024-03-25T19:48:44.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed17fff640 10 monclient: _check_auth_tickets 2024-03-25T19:48:44.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:44.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed35c25640 5 monclient: authenticate success, global_id 4539 2024-03-25T19:48:44.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed35c25640 10 monclient: _renew_subs 2024-03-25T19:48:44.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed35c25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:44.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed35c25640 1 -- 172.21.15.81:0/340392850 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed301316c0 con 0x7fed300b1730 2024-03-25T19:48:44.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed35c25640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:44.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed35c25640 10 monclient: _renew_subs 2024-03-25T19:48:44.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed35c25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:44.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed35c25640 1 -- 172.21.15.81:0/340392850 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed30137860 con 0x7fed300b1730 2024-03-25T19:48:44.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.155+0000 7fed35c25640 1 librados: init done 2024-03-25T19:48:44.157 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.156+0000 7fed35c25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:44.157 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.156+0000 7fed35c25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:44.157 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.156+0000 7fed35c25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:44.157 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.156+0000 7fed35c25640 1 -- 172.21.15.81:0/340392850 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed30132c40 con 0x7fed300b1730 2024-03-25T19:48:44.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.473+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed200035f0 con 0x7fed300b1730 2024-03-25T19:48:44.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.473+0000 7fed17fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:44.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.473+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fed20003a00 con 0x7fed300b1730 2024-03-25T19:48:44.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.473+0000 7fed17fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:44.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.473+0000 7fed17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:44.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.473+0000 7fed17fff640 10 monclient: dump: 2024-03-25T19:48:44.473 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:44.474 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:44.474 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:44.474 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:44.474 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:44.474 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:44.474 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:44.474 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:44.474 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:44.474 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:44.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.473+0000 7fed2dffb640 4 set_mon_vals no callback set 2024-03-25T19:48:44.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.474+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fed20003c20 con 0x7fed300b1730 2024-03-25T19:48:44.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.475+0000 7fed2effd640 10 monclient: get_auth_request con 0x7fed1c06e160 auth_method 0 2024-03-25T19:48:44.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.475+0000 7fed17fff640 10 client.4539.objecter ms_handle_connect 0x7fed1c06e160 2024-03-25T19:48:44.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.480+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fed200535b0 con 0x7fed300b1730 2024-03-25T19:48:44.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.480+0000 7fed17fff640 10 client.4539.objecter ms_dispatch 0x7fed30000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:44.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.480+0000 7fed17fff640 3 client.4539.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:44.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.480+0000 7fed17fff640 3 client.4539.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:44.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.480+0000 7fed17fff640 20 client.4539.objecter dump_active .. 0 homeless 2024-03-25T19:48:44.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.480+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed2004f0c0 con 0x7fed300b1730 2024-03-25T19:48:44.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.480+0000 7fed17fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:44.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.480+0000 7fed17fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:44.731 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.731+0000 7fed35c25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:44.731 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.731+0000 7fed35c25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:44.731 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.731+0000 7fed35c25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:44.732 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.731+0000 7fed35c25640 1 -- 172.21.15.81:0/340392850 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed300627b0 con 0x7fed300b1730 2024-03-25T19:48:44.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.763+0000 7fed17fff640 1 -- 172.21.15.81:0/340392850 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fed20050ff0 con 0x7fed300b1730 2024-03-25T19:48:44.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.763+0000 7fed17fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:44.763 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.763+0000 7fed17fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:44.763 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:48:44.763 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:44.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.765+0000 7fed35c25640 10 librados: watch_flush enter 2024-03-25T19:48:44.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.765+0000 7fed35c25640 10 librados: watch_flush exit 2024-03-25T19:48:44.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.765+0000 7fed35c25640 20 client.4539.objecter shutdown clearing up homeless session... 2024-03-25T19:48:44.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.765+0000 7fed35c25640 10 client.4539.objecter successfully canceled tick 2024-03-25T19:48:44.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.765+0000 7fed35c25640 1 -- 172.21.15.81:0/340392850 >> v1:172.21.15.98:6800/3980829775 conn(0x7fed1c06e160 legacy=0x7fed1c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:44.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.765+0000 7fed35c25640 10 monclient: shutdown 2024-03-25T19:48:44.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.765+0000 7fed35c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:44.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.765+0000 7fed35c25640 1 -- 172.21.15.81:0/340392850 >> v1:172.21.15.81:6789/0 conn(0x7fed300b1730 legacy=0x7fed301393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:44.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.765+0000 7fed35c25640 1 -- 172.21.15.81:0/340392850 shutdown_connections 2024-03-25T19:48:44.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.765+0000 7fed35c25640 1 -- 172.21.15.81:0/340392850 >> 172.21.15.81:0/340392850 conn(0x7fed300617a0 msgr2=0x7fed3013c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:44.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.765+0000 7fed35c25640 1 -- 172.21.15.81:0/340392850 shutdown_connections 2024-03-25T19:48:44.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.765+0000 7fed35c25640 1 -- 172.21.15.81:0/340392850 wait complete. 2024-03-25T19:48:44.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:44.765+0000 7fed35c25640 1 librados: shutdown 2024-03-25T19:48:44.777 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:51.584 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:51.682 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.682+0000 7f6338c58640 10 monclient: get_monmap_and_config 2024-03-25T19:48:51.682 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.682+0000 7f6338c58640 10 monclient: build_initial_monmap 2024-03-25T19:48:51.682 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.682+0000 7f6338c58640 10 monclient: monmap: 2024-03-25T19:48:51.682 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:48:51.683 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:51.683 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:48:51.683632+0000 2024-03-25T19:48:51.683 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:48:51.683632+0000 2024-03-25T19:48:51.683 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:51.683 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:51.683 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:48:51.683 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:48:51.683 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:48:51.683 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:51.684 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.683+0000 7f6338c58640 1 Processor -- start 2024-03-25T19:48:51.684 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.683+0000 7f6338c58640 1 -- start start 2024-03-25T19:48:51.684 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.683+0000 7f6338c58640 10 monclient: init 2024-03-25T19:48:51.684 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.684+0000 7f6338c58640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:51.685 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.684+0000 7f6338c58640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:51.685 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.684+0000 7f6338c58640 10 monclient(hunting): picked mon.noname-a con 0x7f63340a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:48:51.685 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.684+0000 7f6338c58640 10 monclient(hunting): picked mon.noname-c con 0x7f633413f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:48:51.685 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.684+0000 7f6338c58640 10 monclient(hunting): picked mon.noname-b con 0x7f63341430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:48:51.685 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.684+0000 7f6338c58640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6334142520 con 0x7f63340a98f0 2024-03-25T19:48:51.685 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.684+0000 7f6338c58640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63340b28a0 con 0x7f633413f710 2024-03-25T19:48:51.685 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.684+0000 7f6338c58640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63340b2a30 con 0x7f63341430c0 2024-03-25T19:48:51.685 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.684+0000 7f6338c58640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:51.685 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.684+0000 7f6338c58640 10 monclient(hunting): authenticate will time out at 4366.766431s 2024-03-25T19:48:51.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.694+0000 7f632bfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f633413f710 0x7f63341429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:42264/0 (socket says 172.21.15.81:42264) 2024-03-25T19:48:51.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.694+0000 7f632bfff640 1 -- 172.21.15.81:0/2376965970 learned_addr learned my addr 172.21.15.81:0/2376965970 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:48:51.720 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.721+0000 7f6331d9b640 1 -- 172.21.15.81:0/2376965970 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360951928 0 0) 0x7f6334142520 con 0x7f63340a98f0 2024-03-25T19:48:51.721 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.721+0000 7f6331d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:48:51.721 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.721+0000 7f6331d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:51.721 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.721+0000 7f6331d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:51.721 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.721+0000 7f6331d9b640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:48:51.721 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.721+0000 7f6331d9b640 1 -- 172.21.15.81:0/2376965970 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6320003520 con 0x7f63340a98f0 2024-03-25T19:48:51.737 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.737+0000 7f6331d9b640 1 -- 172.21.15.81:0/2376965970 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2325869939 0 0) 0x7f6320003520 con 0x7f63340a98f0 2024-03-25T19:48:51.737 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.737+0000 7f6331d9b640 1 -- 172.21.15.81:0/2376965970 >> v1:172.21.15.81:6790/0 conn(0x7f63341430c0 legacy=0x7f633414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:48:51.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.737+0000 7f6331d9b640 1 -- 172.21.15.81:0/2376965970 >> v1:172.21.15.98:6789/0 conn(0x7f633413f710 legacy=0x7f63341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:51.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.737+0000 7f6331d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:48:51.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.737+0000 7f6331d9b640 1 monclient: found mon.noname-a 2024-03-25T19:48:51.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.737+0000 7f6331d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:51.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.737+0000 7f6331d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:48:51.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.737+0000 7f6331d9b640 1 -- 172.21.15.81:0/2376965970 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63340b44c0 con 0x7f63340a98f0 2024-03-25T19:48:51.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.737+0000 7f6331d9b640 10 monclient: _finish_auth 0 2024-03-25T19:48:51.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.737+0000 7f6331d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:48:51.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.737+0000 7f6331d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:51.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.737+0000 7f6331d9b640 1 -- 172.21.15.81:0/2376965970 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f6324001690 con 0x7f63340a98f0 2024-03-25T19:48:51.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.737+0000 7f6331d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:51.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.737+0000 7f6331d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:51.738 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.737+0000 7f6331d9b640 10 monclient: dump: 2024-03-25T19:48:51.738 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:51.739 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:51.739 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:51.739 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:51.739 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:51.739 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:51.739 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:51.739 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:51.739 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:51.739 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:51.739 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.737+0000 7f6338c58640 5 monclient: authenticate success, global_id 4542 2024-03-25T19:48:51.739 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.737+0000 7f6338c58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:51.758 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.758+0000 7f6331d9b640 1 -- 172.21.15.81:0/2376965970 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6324001860 con 0x7f63340a98f0 2024-03-25T19:48:51.758 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.758+0000 7f6331d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:51.758 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.758+0000 7f6331d9b640 1 -- 172.21.15.81:0/2376965970 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f6324001cb0 con 0x7f63340a98f0 2024-03-25T19:48:51.758 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.758+0000 7f6338c58640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:51.758 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.758+0000 7f6338c58640 4 set_mon_vals no callback set 2024-03-25T19:48:51.758 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.758+0000 7f6331d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:51.758 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.758+0000 7f6331d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:51.759 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.758+0000 7f6331d9b640 10 monclient: dump: 2024-03-25T19:48:51.759 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:51.759 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:51.759 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:51.759 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:51.759 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:51.759 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:51.759 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:51.759 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:51.759 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:51.759 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:51.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.759+0000 7f6338c58640 10 monclient: shutdown 2024-03-25T19:48:51.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.759+0000 7f6338c58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:51.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.759+0000 7f6338c58640 1 -- 172.21.15.81:0/2376965970 >> v1:172.21.15.81:6789/0 conn(0x7f63340a98f0 legacy=0x7f63340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:51.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.759+0000 7f6338c58640 1 -- 172.21.15.81:0/2376965970 shutdown_connections 2024-03-25T19:48:51.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.759+0000 7f6338c58640 1 -- 172.21.15.81:0/2376965970 >> 172.21.15.81:0/2376965970 conn(0x7f63340b7ea0 msgr2=0x7f63340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:51.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.759+0000 7f6338c58640 1 -- 172.21.15.81:0/2376965970 shutdown_connections 2024-03-25T19:48:51.760 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.760+0000 7f6338c58640 1 -- 172.21.15.81:0/2376965970 wait complete. 2024-03-25T19:48:51.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.761+0000 7f6338c58640 10 monclient: build_initial_monmap 2024-03-25T19:48:51.761 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.761+0000 7f6338c58640 10 monclient: monmap: 2024-03-25T19:48:51.761 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:48:51.761 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:51.761 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:48:51.761 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:48:51.761 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:51.761 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:48:51.762 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:48:51.762 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:48:51.762 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:48:51.762 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:51.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.761+0000 7f6338c58640 1 librados: starting msgr at 2024-03-25T19:48:51.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.761+0000 7f6338c58640 1 librados: starting objecter 2024-03-25T19:48:51.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.761+0000 7f6338c58640 1 Processor -- start 2024-03-25T19:48:51.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.761+0000 7f6338c58640 1 -- start start 2024-03-25T19:48:51.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.761+0000 7f6338c58640 1 librados: setting wanted keys 2024-03-25T19:48:51.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.761+0000 7f6338c58640 1 librados: calling monclient init 2024-03-25T19:48:51.762 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.761+0000 7f6338c58640 10 monclient: init 2024-03-25T19:48:51.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.762+0000 7f6338c58640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:51.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.762+0000 7f6338c58640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:51.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.762+0000 7f6338c58640 10 monclient(hunting): picked mon.noname-a con 0x7f63340a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:48:51.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.762+0000 7f6338c58640 10 monclient(hunting): picked mon.noname-c con 0x7f633413f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:48:51.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.762+0000 7f6338c58640 10 monclient(hunting): picked mon.noname-b con 0x7f63341430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:48:51.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.762+0000 7f6338c58640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6334070770 con 0x7f63340a98f0 2024-03-25T19:48:51.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.762+0000 7f6338c58640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f633406f4a0 con 0x7f63341430c0 2024-03-25T19:48:51.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.762+0000 7f6338c58640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6334070b90 con 0x7f633413f710 2024-03-25T19:48:51.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.762+0000 7f6338c58640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:51.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.763+0000 7f6338c58640 10 monclient(hunting): authenticate will time out at 4366.844850s 2024-03-25T19:48:51.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.764+0000 7f6332d9d640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f63341430c0 0x7f6334138fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:42310/0 (socket says 172.21.15.81:42310) 2024-03-25T19:48:51.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.764+0000 7f6332d9d640 1 -- 172.21.15.81:0/2881200907 learned_addr learned my addr 172.21.15.81:0/2881200907 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:48:51.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.765+0000 7f632affd640 10 client.?.objecter ms_handle_connect 0x7f63341430c0 2024-03-25T19:48:51.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.765+0000 7f632affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:51.784 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.784+0000 7f632affd640 10 client.?.objecter ms_handle_connect 0x7f633413f710 2024-03-25T19:48:51.784 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.784+0000 7f632affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:51.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.819+0000 7f632affd640 10 client.?.objecter ms_handle_connect 0x7f63340a98f0 2024-03-25T19:48:51.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.819+0000 7f632affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:51.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.823+0000 7f632affd640 1 -- 172.21.15.81:0/2881200907 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3361690770 0 0) 0x7f6334070770 con 0x7f63340a98f0 2024-03-25T19:48:51.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.823+0000 7f632affd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:51.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.823+0000 7f632affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:51.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.823+0000 7f632affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:51.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.823+0000 7f632affd640 10 monclient(hunting): my global_id is 4545 2024-03-25T19:48:51.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.823+0000 7f632affd640 1 -- 172.21.15.81:0/2881200907 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6314004eb0 con 0x7f63340a98f0 2024-03-25T19:48:51.835 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.835+0000 7f632affd640 1 -- 172.21.15.81:0/2881200907 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1915178547 0 0) 0x7f6314004eb0 con 0x7f63340a98f0 2024-03-25T19:48:51.835 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.836+0000 7f632affd640 1 -- 172.21.15.81:0/2881200907 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6334070770 con 0x7f63340a98f0 2024-03-25T19:48:51.835 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.836+0000 7f632affd640 1 -- 172.21.15.81:0/2881200907 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f6324001b70 con 0x7f63340a98f0 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.836+0000 7f632affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.836+0000 7f632affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.836+0000 7f632affd640 10 monclient(hunting): dump: 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:51.836 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:51.837 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.836+0000 7f632affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:51.838 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f632affd640 1 -- 172.21.15.81:0/2881200907 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4037755181 0 0) 0x7f6334070770 con 0x7f63340a98f0 2024-03-25T19:48:51.838 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f632affd640 1 -- 172.21.15.81:0/2881200907 >> v1:172.21.15.81:6790/0 conn(0x7f633413f710 legacy=0x7f63341dbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:51.838 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f632affd640 1 -- 172.21.15.81:0/2881200907 >> v1:172.21.15.98:6789/0 conn(0x7f63341430c0 legacy=0x7f6334138fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f632affd640 10 monclient: _finish_hunting 0 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f632affd640 1 monclient: found mon.a 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f632affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f632affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f632affd640 1 -- 172.21.15.81:0/2881200907 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6334070d20 con 0x7f63340a98f0 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f632affd640 10 monclient: _finish_auth 0 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f632affd640 10 monclient: _check_auth_tickets 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f632affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f6338c58640 5 monclient: authenticate success, global_id 4545 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f6338c58640 10 monclient: _renew_subs 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f6338c58640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f6338c58640 1 -- 172.21.15.81:0/2881200907 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63341dc6c0 con 0x7f63340a98f0 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f6338c58640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:51.839 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f6338c58640 10 monclient: _renew_subs 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f6338c58640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f6338c58640 1 -- 172.21.15.81:0/2881200907 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f633406e7b0 con 0x7f63340a98f0 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.838+0000 7f6338c58640 1 librados: init done 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.839+0000 7f6338c58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.839+0000 7f6338c58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.839+0000 7f6338c58640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:51.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:51.839+0000 7f6338c58640 1 -- 172.21.15.81:0/2881200907 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f633413def0 con 0x7f63340a98f0 2024-03-25T19:48:52.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.152+0000 7f632affd640 1 -- 172.21.15.81:0/2881200907 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6324001cd0 con 0x7f63340a98f0 2024-03-25T19:48:52.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.152+0000 7f632affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:52.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.152+0000 7f632affd640 1 -- 172.21.15.81:0/2881200907 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f6324003b00 con 0x7f63340a98f0 2024-03-25T19:48:52.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.152+0000 7f632affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:52.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.152+0000 7f632affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:52.153 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.152+0000 7f632affd640 10 monclient: dump: 2024-03-25T19:48:52.153 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:52.153 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:52.153 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:52.153 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:52.153 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:52.153 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:52.153 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:52.153 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:52.153 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:52.153 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:52.153 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.152+0000 7f633159a640 4 set_mon_vals no callback set 2024-03-25T19:48:52.154 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.153+0000 7f632affd640 1 -- 172.21.15.81:0/2881200907 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f6324019b10 con 0x7f63340a98f0 2024-03-25T19:48:52.154 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.154+0000 7f632bfff640 10 monclient: get_auth_request con 0x7f631406dc40 auth_method 0 2024-03-25T19:48:52.154 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.154+0000 7f632affd640 10 client.4545.objecter ms_handle_connect 0x7f631406dc40 2024-03-25T19:48:52.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.159+0000 7f632affd640 1 -- 172.21.15.81:0/2881200907 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f63240537e0 con 0x7f63340a98f0 2024-03-25T19:48:52.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.159+0000 7f632affd640 10 client.4545.objecter ms_dispatch 0x7f6334000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:52.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.159+0000 7f632affd640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:52.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.159+0000 7f632affd640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:52.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.159+0000 7f632affd640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-25T19:48:52.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.159+0000 7f632affd640 1 -- 172.21.15.81:0/2881200907 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f632404f120 con 0x7f63340a98f0 2024-03-25T19:48:52.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.159+0000 7f632affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:52.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.159+0000 7f632affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:52.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.405+0000 7f6338c58640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:52.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.405+0000 7f6338c58640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:52.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.405+0000 7f6338c58640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:52.405 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.405+0000 7f6338c58640 1 -- 172.21.15.81:0/2881200907 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63340aa830 con 0x7f63340a98f0 2024-03-25T19:48:52.435 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.436+0000 7f632affd640 1 -- 172.21.15.81:0/2881200907 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f6324051220 con 0x7f63340a98f0 2024-03-25T19:48:52.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.436+0000 7f632affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:52.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.436+0000 7f632affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:52.436 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:48:52.436 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:52.438 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.438+0000 7f6338c58640 10 librados: watch_flush enter 2024-03-25T19:48:52.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.438+0000 7f6338c58640 10 librados: watch_flush exit 2024-03-25T19:48:52.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.438+0000 7f6338c58640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-25T19:48:52.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.438+0000 7f6338c58640 10 client.4545.objecter successfully canceled tick 2024-03-25T19:48:52.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.438+0000 7f6338c58640 1 -- 172.21.15.81:0/2881200907 >> v1:172.21.15.98:6800/3980829775 conn(0x7f631406dc40 legacy=0x7f6314070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:52.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.438+0000 7f6338c58640 10 monclient: shutdown 2024-03-25T19:48:52.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.438+0000 7f6338c58640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:52.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.438+0000 7f6338c58640 1 -- 172.21.15.81:0/2881200907 >> v1:172.21.15.81:6789/0 conn(0x7f63340a98f0 legacy=0x7f633406ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:52.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.438+0000 7f6338c58640 1 -- 172.21.15.81:0/2881200907 shutdown_connections 2024-03-25T19:48:52.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.438+0000 7f6338c58640 1 -- 172.21.15.81:0/2881200907 >> 172.21.15.81:0/2881200907 conn(0x7f63340b7ea0 msgr2=0x7f6334074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:52.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.438+0000 7f6338c58640 1 -- 172.21.15.81:0/2881200907 shutdown_connections 2024-03-25T19:48:52.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.438+0000 7f6338c58640 1 -- 172.21.15.81:0/2881200907 wait complete. 2024-03-25T19:48:52.439 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:52.439+0000 7f6338c58640 1 librados: shutdown 2024-03-25T19:48:52.449 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:59.256 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:59.370 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.370+0000 7f4c55aba640 10 monclient: get_monmap_and_config 2024-03-25T19:48:59.370 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.370+0000 7f4c55aba640 10 monclient: build_initial_monmap 2024-03-25T19:48:59.370 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.370+0000 7f4c55aba640 10 monclient: monmap: 2024-03-25T19:48:59.370 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:48:59.371 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:59.371 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:48:59.371773+0000 2024-03-25T19:48:59.371 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:48:59.371773+0000 2024-03-25T19:48:59.371 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:59.371 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:59.371 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:48:59.371 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:48:59.371 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:48:59.371 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:59.372 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.371+0000 7f4c55aba640 1 Processor -- start 2024-03-25T19:48:59.372 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.371+0000 7f4c55aba640 1 -- start start 2024-03-25T19:48:59.372 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.371+0000 7f4c55aba640 10 monclient: init 2024-03-25T19:48:59.372 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.372+0000 7f4c55aba640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:59.372 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.372+0000 7f4c55aba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:59.372 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.372+0000 7f4c55aba640 10 monclient(hunting): picked mon.noname-c con 0x7f4c500b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:48:59.372 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.372+0000 7f4c55aba640 10 monclient(hunting): picked mon.noname-a con 0x7f4c500b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:48:59.372 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.372+0000 7f4c55aba640 10 monclient(hunting): picked mon.noname-b con 0x7f4c5014be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:48:59.373 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.372+0000 7f4c55aba640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c500af940 con 0x7f4c500b1730 2024-03-25T19:48:59.373 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.372+0000 7f4c55aba640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c500b0a70 con 0x7f4c500b4d60 2024-03-25T19:48:59.373 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.372+0000 7f4c55aba640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c500b0d50 con 0x7f4c5014be90 2024-03-25T19:48:59.373 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.372+0000 7f4c55aba640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:59.373 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.372+0000 7f4c55aba640 10 monclient(hunting): authenticate will time out at 4374.454437s 2024-03-25T19:48:59.377 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.377+0000 7f4c4f7fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4c500b4d60 0x7f4c5013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:44916/0 (socket says 172.21.15.81:44916) 2024-03-25T19:48:59.377 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.377+0000 7f4c4f7fe640 1 -- 172.21.15.81:0/2043683327 learned_addr learned my addr 172.21.15.81:0/2043683327 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:48:59.390 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.390+0000 7f4c4e7fc640 1 -- 172.21.15.81:0/2043683327 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421278952 0 0) 0x7f4c500b0a70 con 0x7f4c500b4d60 2024-03-25T19:48:59.390 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.390+0000 7f4c4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:59.390 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.390+0000 7f4c4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:59.390 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.390+0000 7f4c4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:59.390 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.390+0000 7f4c4e7fc640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:48:59.390 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.390+0000 7f4c4e7fc640 1 -- 172.21.15.81:0/2043683327 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c38003520 con 0x7f4c500b4d60 2024-03-25T19:48:59.392 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.393+0000 7f4c4e7fc640 1 -- 172.21.15.81:0/2043683327 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719112265 0 0) 0x7f4c500af940 con 0x7f4c500b1730 2024-03-25T19:48:59.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.393+0000 7f4c4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:59.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.393+0000 7f4c4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:59.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.393+0000 7f4c4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:59.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.393+0000 7f4c4e7fc640 10 monclient(hunting): my global_id is 4548 2024-03-25T19:48:59.393 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.393+0000 7f4c4e7fc640 1 -- 172.21.15.81:0/2043683327 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c500b0a70 con 0x7f4c500b1730 2024-03-25T19:48:59.398 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.399+0000 7f4c4e7fc640 1 -- 172.21.15.81:0/2043683327 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043193572 0 0) 0x7f4c500b0d50 con 0x7f4c5014be90 2024-03-25T19:48:59.398 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.399+0000 7f4c4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:59.398 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.399+0000 7f4c4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:59.398 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.399+0000 7f4c4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:59.399 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.399+0000 7f4c4e7fc640 10 monclient(hunting): my global_id is 4517 2024-03-25T19:48:59.399 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.399+0000 7f4c4e7fc640 1 -- 172.21.15.81:0/2043683327 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c500af940 con 0x7f4c5014be90 2024-03-25T19:48:59.406 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.406+0000 7f4c4e7fc640 1 -- 172.21.15.81:0/2043683327 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3473802315 0 0) 0x7f4c38003520 con 0x7f4c500b4d60 2024-03-25T19:48:59.406 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.406+0000 7f4c4e7fc640 1 -- 172.21.15.81:0/2043683327 >> v1:172.21.15.81:6790/0 conn(0x7f4c5014be90 legacy=0x7f4c5014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:59.407 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.406+0000 7f4c4e7fc640 1 -- 172.21.15.81:0/2043683327 >> v1:172.21.15.81:6789/0 conn(0x7f4c500b1730 legacy=0x7f4c500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:59.407 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.406+0000 7f4c4e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:48:59.407 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.406+0000 7f4c4e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:48:59.407 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.406+0000 7f4c4e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:59.407 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.406+0000 7f4c4e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:48:59.407 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.406+0000 7f4c4e7fc640 1 -- 172.21.15.81:0/2043683327 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c500b1030 con 0x7f4c500b4d60 2024-03-25T19:48:59.407 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.406+0000 7f4c4e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:48:59.407 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.406+0000 7f4c4e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:48:59.407 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.406+0000 7f4c4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:59.407 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.406+0000 7f4c4e7fc640 1 -- 172.21.15.81:0/2043683327 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4c40001690 con 0x7f4c500b4d60 2024-03-25T19:48:59.407 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.406+0000 7f4c4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:59.407 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.406+0000 7f4c4e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:59.407 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.406+0000 7f4c4e7fc640 10 monclient: dump: 2024-03-25T19:48:59.407 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:59.408 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:59.408 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:59.408 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:59.408 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:59.408 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:59.408 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:59.408 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:59.408 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:59.408 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:59.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.406+0000 7f4c55aba640 5 monclient: authenticate success, global_id 4534 2024-03-25T19:48:59.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.406+0000 7f4c55aba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:59.415 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.416+0000 7f4c4e7fc640 1 -- 172.21.15.81:0/2043683327 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c400017f0 con 0x7f4c500b4d60 2024-03-25T19:48:59.415 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.416+0000 7f4c4e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:59.415 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.416+0000 7f4c55aba640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:59.415 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.416+0000 7f4c55aba640 4 set_mon_vals no callback set 2024-03-25T19:48:59.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.417+0000 7f4c55aba640 10 monclient: shutdown 2024-03-25T19:48:59.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.417+0000 7f4c55aba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:59.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.417+0000 7f4c55aba640 1 -- 172.21.15.81:0/2043683327 >> v1:172.21.15.98:6789/0 conn(0x7f4c500b4d60 legacy=0x7f4c5013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:59.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.417+0000 7f4c55aba640 1 -- 172.21.15.81:0/2043683327 shutdown_connections 2024-03-25T19:48:59.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.417+0000 7f4c55aba640 1 -- 172.21.15.81:0/2043683327 >> 172.21.15.81:0/2043683327 conn(0x7f4c500617a0 msgr2=0x7f4c50061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:59.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.417+0000 7f4c55aba640 1 -- 172.21.15.81:0/2043683327 shutdown_connections 2024-03-25T19:48:59.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.417+0000 7f4c55aba640 1 -- 172.21.15.81:0/2043683327 wait complete. 2024-03-25T19:48:59.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.418+0000 7f4c55aba640 10 monclient: build_initial_monmap 2024-03-25T19:48:59.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.418+0000 7f4c55aba640 10 monclient: monmap: 2024-03-25T19:48:59.417 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:48:59.418 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:59.418 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:48:59.418 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:48:59.418 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:59.418 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:48:59.418 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:48:59.418 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:48:59.418 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:48:59.418 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:59.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.418+0000 7f4c55aba640 1 librados: starting msgr at 2024-03-25T19:48:59.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.418+0000 7f4c55aba640 1 librados: starting objecter 2024-03-25T19:48:59.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.418+0000 7f4c55aba640 1 Processor -- start 2024-03-25T19:48:59.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.419+0000 7f4c55aba640 1 -- start start 2024-03-25T19:48:59.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.419+0000 7f4c55aba640 1 librados: setting wanted keys 2024-03-25T19:48:59.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.419+0000 7f4c55aba640 1 librados: calling monclient init 2024-03-25T19:48:59.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.419+0000 7f4c55aba640 10 monclient: init 2024-03-25T19:48:59.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.419+0000 7f4c55aba640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:59.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.419+0000 7f4c55aba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:59.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.419+0000 7f4c55aba640 10 monclient(hunting): picked mon.noname-c con 0x7f4c500b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:48:59.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.419+0000 7f4c55aba640 10 monclient(hunting): picked mon.noname-a con 0x7f4c500b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:48:59.420 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.419+0000 7f4c55aba640 10 monclient(hunting): picked mon.noname-b con 0x7f4c5014be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:48:59.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.419+0000 7f4c55aba640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c501dae70 con 0x7f4c500b4d60 2024-03-25T19:48:59.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.419+0000 7f4c55aba640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c501dba10 con 0x7f4c5014be90 2024-03-25T19:48:59.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.419+0000 7f4c55aba640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c50140bc0 con 0x7f4c500b1730 2024-03-25T19:48:59.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.419+0000 7f4c55aba640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:59.421 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.419+0000 7f4c55aba640 10 monclient(hunting): authenticate will time out at 4374.501568s 2024-03-25T19:48:59.427 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.427+0000 7f4c4effd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f4c500b4d60 0x7f4c5013a360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:53682/0 (socket says 172.21.15.81:53682) 2024-03-25T19:48:59.427 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.428+0000 7f4c4effd640 1 -- 172.21.15.81:0/2638871976 learned_addr learned my addr 172.21.15.81:0/2638871976 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:48:59.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.428+0000 7f4c37fff640 10 client.?.objecter ms_handle_connect 0x7f4c500b4d60 2024-03-25T19:48:59.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.428+0000 7f4c37fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:59.430 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.430+0000 7f4c37fff640 1 -- 172.21.15.81:0/2638871976 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050126647 0 0) 0x7f4c501dae70 con 0x7f4c500b4d60 2024-03-25T19:48:59.430 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.430+0000 7f4c37fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:59.430 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.430+0000 7f4c37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:59.430 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.430+0000 7f4c37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:59.430 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.430+0000 7f4c37fff640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:48:59.430 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.430+0000 7f4c37fff640 1 -- 172.21.15.81:0/2638871976 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c3c004eb0 con 0x7f4c500b4d60 2024-03-25T19:48:59.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.434+0000 7f4c37fff640 10 client.?.objecter ms_handle_connect 0x7f4c5014be90 2024-03-25T19:48:59.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.434+0000 7f4c37fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:59.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.434+0000 7f4c37fff640 10 client.?.objecter ms_handle_connect 0x7f4c500b1730 2024-03-25T19:48:59.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.434+0000 7f4c37fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:59.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.441+0000 7f4c37fff640 1 -- 172.21.15.81:0/2638871976 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2947691676 0 0) 0x7f4c3c004eb0 con 0x7f4c500b4d60 2024-03-25T19:48:59.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.441+0000 7f4c37fff640 1 -- 172.21.15.81:0/2638871976 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c501dae70 con 0x7f4c500b4d60 2024-03-25T19:48:59.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.441+0000 7f4c37fff640 1 -- 172.21.15.81:0/2638871976 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4c44001a30 con 0x7f4c500b4d60 2024-03-25T19:48:59.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.441+0000 7f4c37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:59.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.441+0000 7f4c37fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:59.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.441+0000 7f4c37fff640 10 monclient(hunting): dump: 2024-03-25T19:48:59.441 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:59.441 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:59.442 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:59.442 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:59.442 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:59.442 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:59.442 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:59.442 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:59.442 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:59.442 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:59.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.441+0000 7f4c37fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:59.446 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.446+0000 7f4c37fff640 1 -- 172.21.15.81:0/2638871976 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621903864 0 0) 0x7f4c501dba10 con 0x7f4c5014be90 2024-03-25T19:48:59.446 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.447+0000 7f4c37fff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:59.446 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.447+0000 7f4c37fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:59.446 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.447+0000 7f4c37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:59.446 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.447+0000 7f4c37fff640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:48:59.447 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.447+0000 7f4c37fff640 1 -- 172.21.15.81:0/2638871976 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c3c004eb0 con 0x7f4c5014be90 2024-03-25T19:48:59.449 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c37fff640 1 -- 172.21.15.81:0/2638871976 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 592651700 0 0) 0x7f4c501dae70 con 0x7f4c500b4d60 2024-03-25T19:48:59.449 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c37fff640 1 -- 172.21.15.81:0/2638871976 >> v1:172.21.15.81:6790/0 conn(0x7f4c500b1730 legacy=0x7f4c501db300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:59.449 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c37fff640 1 -- 172.21.15.81:0/2638871976 >> v1:172.21.15.98:6789/0 conn(0x7f4c5014be90 legacy=0x7f4c501e2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:59.449 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c37fff640 10 monclient: _finish_hunting 0 2024-03-25T19:48:59.449 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c37fff640 1 monclient: found mon.a 2024-03-25T19:48:59.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:59.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c37fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:59.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c37fff640 1 -- 172.21.15.81:0/2638871976 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5013b200 con 0x7f4c500b4d60 2024-03-25T19:48:59.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c37fff640 10 monclient: _finish_auth 0 2024-03-25T19:48:59.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c37fff640 10 monclient: _check_auth_tickets 2024-03-25T19:48:59.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:59.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c55aba640 5 monclient: authenticate success, global_id 4551 2024-03-25T19:48:59.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c55aba640 10 monclient: _renew_subs 2024-03-25T19:48:59.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c55aba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:59.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c55aba640 1 -- 172.21.15.81:0/2638871976 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c5013aa70 con 0x7f4c500b4d60 2024-03-25T19:48:59.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c55aba640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:59.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c55aba640 10 monclient: _renew_subs 2024-03-25T19:48:59.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c55aba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:59.450 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c55aba640 1 -- 172.21.15.81:0/2638871976 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c5013bd10 con 0x7f4c500b4d60 2024-03-25T19:48:59.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.449+0000 7f4c55aba640 1 librados: init done 2024-03-25T19:48:59.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.450+0000 7f4c55aba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:59.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.450+0000 7f4c55aba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:59.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.450+0000 7f4c55aba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:48:59.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.450+0000 7f4c55aba640 1 -- 172.21.15.81:0/2638871976 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c501dac20 con 0x7f4c500b4d60 2024-03-25T19:48:59.787 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.787+0000 7f4c37fff640 1 -- 172.21.15.81:0/2638871976 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c44001cd0 con 0x7f4c500b4d60 2024-03-25T19:48:59.787 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.787+0000 7f4c37fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:59.788 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.787+0000 7f4c37fff640 1 -- 172.21.15.81:0/2638871976 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4c44003a60 con 0x7f4c500b4d60 2024-03-25T19:48:59.788 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.787+0000 7f4c37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:59.788 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.787+0000 7f4c37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:59.788 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.787+0000 7f4c37fff640 10 monclient: dump: 2024-03-25T19:48:59.788 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:48:59.788 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:48:59.788 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:59.788 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:48:59.788 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:59.788 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:48:59.788 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:48:59.788 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:48:59.788 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:48:59.788 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:48:59.789 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.787+0000 7f4c4dffb640 4 set_mon_vals no callback set 2024-03-25T19:48:59.789 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.788+0000 7f4c37fff640 1 -- 172.21.15.81:0/2638871976 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f4c44003c80 con 0x7f4c500b4d60 2024-03-25T19:48:59.789 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.789+0000 7f4c4f7fe640 10 monclient: get_auth_request con 0x7f4c3c06dd90 auth_method 0 2024-03-25T19:48:59.789 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.789+0000 7f4c37fff640 10 client.4551.objecter ms_handle_connect 0x7f4c3c06dd90 2024-03-25T19:48:59.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.797+0000 7f4c37fff640 1 -- 172.21.15.81:0/2638871976 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f4c44054860 con 0x7f4c500b4d60 2024-03-25T19:48:59.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.797+0000 7f4c37fff640 10 client.4551.objecter ms_dispatch 0x7f4c50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:59.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.797+0000 7f4c37fff640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:59.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.797+0000 7f4c37fff640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:59.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.797+0000 7f4c37fff640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-25T19:48:59.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.797+0000 7f4c37fff640 1 -- 172.21.15.81:0/2638871976 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c44050370 con 0x7f4c500b4d60 2024-03-25T19:48:59.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.797+0000 7f4c37fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:59.797 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:48:59.797+0000 7f4c37fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:00.008 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.008+0000 7f4c55aba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:00.008 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.008+0000 7f4c55aba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:00.008 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.008+0000 7f4c55aba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:49:00.008 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.008+0000 7f4c55aba640 1 -- 172.21.15.81:0/2638871976 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c500627b0 con 0x7f4c500b4d60 2024-03-25T19:49:00.026 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.026+0000 7f4c37fff640 1 -- 172.21.15.81:0/2638871976 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f4c440522a0 con 0x7f4c500b4d60 2024-03-25T19:49:00.026 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.026+0000 7f4c37fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:00.026 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.026+0000 7f4c37fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:00.026 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:49:00.026 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:00.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.028+0000 7f4c55aba640 10 librados: watch_flush enter 2024-03-25T19:49:00.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.028+0000 7f4c55aba640 10 librados: watch_flush exit 2024-03-25T19:49:00.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.028+0000 7f4c55aba640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-25T19:49:00.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.028+0000 7f4c55aba640 10 client.4551.objecter successfully canceled tick 2024-03-25T19:49:00.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.028+0000 7f4c55aba640 1 -- 172.21.15.81:0/2638871976 >> v1:172.21.15.98:6800/3980829775 conn(0x7f4c3c06dd90 legacy=0x7f4c3c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:00.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.028+0000 7f4c55aba640 10 monclient: shutdown 2024-03-25T19:49:00.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.028+0000 7f4c55aba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:00.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.028+0000 7f4c55aba640 1 -- 172.21.15.81:0/2638871976 >> v1:172.21.15.81:6789/0 conn(0x7f4c500b4d60 legacy=0x7f4c5013a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:00.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.028+0000 7f4c55aba640 1 -- 172.21.15.81:0/2638871976 shutdown_connections 2024-03-25T19:49:00.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.028+0000 7f4c55aba640 1 -- 172.21.15.81:0/2638871976 >> 172.21.15.81:0/2638871976 conn(0x7f4c500617a0 msgr2=0x7f4c5013c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:00.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.028+0000 7f4c55aba640 1 -- 172.21.15.81:0/2638871976 shutdown_connections 2024-03-25T19:49:00.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.028+0000 7f4c55aba640 1 -- 172.21.15.81:0/2638871976 wait complete. 2024-03-25T19:49:00.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:00.029+0000 7f4c55aba640 1 librados: shutdown 2024-03-25T19:49:00.039 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:02.866 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:07:25.508 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:02.871 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:07:25.513 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:02.906 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:07:25.548 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:02.914 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:07:25.556 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:02.924 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:25.559 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:02.927 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:25.561 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:02.927 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:07:25.569 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:02.957 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:25.592 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:02.967 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:07:25.608 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:06.846 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:06.963 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.963+0000 7fb5e7dd4640 10 monclient: get_monmap_and_config 2024-03-25T19:49:06.963 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.963+0000 7fb5e7dd4640 10 monclient: build_initial_monmap 2024-03-25T19:49:06.963 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.963+0000 7fb5e7dd4640 10 monclient: monmap: 2024-03-25T19:49:06.964 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:49:06.964 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:06.964 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:49:06.964827+0000 2024-03-25T19:49:06.964 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:49:06.964827+0000 2024-03-25T19:49:06.964 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:06.964 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:06.964 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:49:06.964 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:49:06.964 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:49:06.964 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:06.965 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.964+0000 7fb5e7dd4640 1 Processor -- start 2024-03-25T19:49:06.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.964+0000 7fb5e7dd4640 1 -- start start 2024-03-25T19:49:06.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.964+0000 7fb5e7dd4640 10 monclient: init 2024-03-25T19:49:06.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.965+0000 7fb5e7dd4640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:06.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.965+0000 7fb5e7dd4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:06.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.965+0000 7fb5e7dd4640 10 monclient(hunting): picked mon.noname-a con 0x7fb5e00b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:49:06.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.965+0000 7fb5e7dd4640 10 monclient(hunting): picked mon.noname-b con 0x7fb5e00b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:49:06.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.965+0000 7fb5e7dd4640 10 monclient(hunting): picked mon.noname-c con 0x7fb5e014be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:49:06.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.965+0000 7fb5e7dd4640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e00af940 con 0x7fb5e00b4d60 2024-03-25T19:49:06.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.965+0000 7fb5e7dd4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e00b0a70 con 0x7fb5e014be90 2024-03-25T19:49:06.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.965+0000 7fb5e7dd4640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e00b0d50 con 0x7fb5e00b1730 2024-03-25T19:49:06.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.965+0000 7fb5e7dd4640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:06.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.965+0000 7fb5e7dd4640 10 monclient(hunting): authenticate will time out at 4382.047353s 2024-03-25T19:49:06.978 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.979+0000 7fb5e536f640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fb5e00b1730 0x7fb5e00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:44352/0 (socket says 172.21.15.81:44352) 2024-03-25T19:49:06.979 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.979+0000 7fb5e536f640 1 -- 172.21.15.81:0/4260681578 learned_addr learned my addr 172.21.15.81:0/4260681578 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:49:06.992 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.993+0000 7fb5e4b6e640 1 -- 172.21.15.81:0/4260681578 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279284838 0 0) 0x7fb5e00b0d50 con 0x7fb5e00b1730 2024-03-25T19:49:06.993 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.993+0000 7fb5e4b6e640 10 monclient(hunting): _negotiate 2024-03-25T19:49:06.993 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.993+0000 7fb5e4b6e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:06.993 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.993+0000 7fb5e4b6e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:06.993 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.993+0000 7fb5e4b6e640 10 monclient(hunting): my global_id is 4523 2024-03-25T19:49:06.993 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:06.993+0000 7fb5e4b6e640 1 -- 172.21.15.81:0/4260681578 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5cc003520 con 0x7fb5e00b1730 2024-03-25T19:49:07.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 1 -- 172.21.15.81:0/4260681578 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203119949 0 0) 0x7fb5e00af940 con 0x7fb5e00b4d60 2024-03-25T19:49:07.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 10 monclient(hunting): _negotiate 2024-03-25T19:49:07.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:07.010 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:07.010 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 10 monclient(hunting): my global_id is 4554 2024-03-25T19:49:07.010 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 1 -- 172.21.15.81:0/4260681578 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5e00b0d50 con 0x7fb5e00b4d60 2024-03-25T19:49:07.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 1 -- 172.21.15.81:0/4260681578 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1995675362 0 0) 0x7fb5cc003520 con 0x7fb5e00b1730 2024-03-25T19:49:07.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 1 -- 172.21.15.81:0/4260681578 >> v1:172.21.15.98:6789/0 conn(0x7fb5e014be90 legacy=0x7fb5e014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:07.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 1 -- 172.21.15.81:0/4260681578 >> v1:172.21.15.81:6789/0 conn(0x7fb5e00b4d60 legacy=0x7fb5e013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:07.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 10 monclient: _finish_hunting 0 2024-03-25T19:49:07.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 1 monclient: found mon.noname-b 2024-03-25T19:49:07.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:07.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:49:07.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 1 -- 172.21.15.81:0/4260681578 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e00b1030 con 0x7fb5e00b1730 2024-03-25T19:49:07.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 10 monclient: _finish_auth 0 2024-03-25T19:49:07.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 10 monclient: _check_auth_tickets 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 1 -- 172.21.15.81:0/4260681578 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb5d0001690 con 0x7fb5e00b1730 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e4b6e640 10 monclient: dump: 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:07.012 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:07.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e7dd4640 5 monclient: authenticate success, global_id 4523 2024-03-25T19:49:07.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.010+0000 7fb5e7dd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:07.021 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.022+0000 7fb5e4b6e640 1 -- 172.21.15.81:0/4260681578 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5d00017f0 con 0x7fb5e00b1730 2024-03-25T19:49:07.022 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.022+0000 7fb5e4b6e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:07.022 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.022+0000 7fb5e4b6e640 1 -- 172.21.15.81:0/4260681578 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb5d0001c00 con 0x7fb5e00b1730 2024-03-25T19:49:07.022 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.022+0000 7fb5e4b6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:07.022 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.022+0000 7fb5e4b6e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:07.022 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.022+0000 7fb5e4b6e640 10 monclient: dump: 2024-03-25T19:49:07.022 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:07.022 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:07.022 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:07.022 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:07.022 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:07.022 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:07.022 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:07.022 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:07.022 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:07.023 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:07.023 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.022+0000 7fb5e7dd4640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:07.023 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.022+0000 7fb5e7dd4640 4 set_mon_vals no callback set 2024-03-25T19:49:07.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.023+0000 7fb5e7dd4640 10 monclient: shutdown 2024-03-25T19:49:07.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.023+0000 7fb5e7dd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:07.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.023+0000 7fb5e7dd4640 1 -- 172.21.15.81:0/4260681578 >> v1:172.21.15.81:6790/0 conn(0x7fb5e00b1730 legacy=0x7fb5e00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:07.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.023+0000 7fb5e7dd4640 1 -- 172.21.15.81:0/4260681578 shutdown_connections 2024-03-25T19:49:07.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.023+0000 7fb5e7dd4640 1 -- 172.21.15.81:0/4260681578 >> 172.21.15.81:0/4260681578 conn(0x7fb5e00617a0 msgr2=0x7fb5e0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:07.025 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.023+0000 7fb5e7dd4640 1 -- 172.21.15.81:0/4260681578 shutdown_connections 2024-03-25T19:49:07.025 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.023+0000 7fb5e7dd4640 1 -- 172.21.15.81:0/4260681578 wait complete. 2024-03-25T19:49:07.025 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.024+0000 7fb5e7dd4640 10 monclient: build_initial_monmap 2024-03-25T19:49:07.025 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.024+0000 7fb5e7dd4640 10 monclient: monmap: 2024-03-25T19:49:07.025 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:49:07.025 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:07.025 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:49:07.025 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:49:07.025 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:07.025 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:49:07.025 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:49:07.025 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:49:07.025 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:49:07.025 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:07.026 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.024+0000 7fb5e7dd4640 1 librados: starting msgr at 2024-03-25T19:49:07.026 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.024+0000 7fb5e7dd4640 1 librados: starting objecter 2024-03-25T19:49:07.026 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.025+0000 7fb5e7dd4640 1 Processor -- start 2024-03-25T19:49:07.026 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.025+0000 7fb5e7dd4640 1 -- start start 2024-03-25T19:49:07.026 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.025+0000 7fb5e7dd4640 1 librados: setting wanted keys 2024-03-25T19:49:07.026 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.025+0000 7fb5e7dd4640 1 librados: calling monclient init 2024-03-25T19:49:07.026 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.025+0000 7fb5e7dd4640 10 monclient: init 2024-03-25T19:49:07.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.025+0000 7fb5e7dd4640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:07.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.025+0000 7fb5e7dd4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:07.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.025+0000 7fb5e7dd4640 10 monclient(hunting): picked mon.noname-b con 0x7fb5e00b1730 addr v1:172.21.15.98:6789/0 2024-03-25T19:49:07.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.025+0000 7fb5e7dd4640 10 monclient(hunting): picked mon.noname-a con 0x7fb5e00b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:49:07.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.025+0000 7fb5e7dd4640 10 monclient(hunting): picked mon.noname-c con 0x7fb5e014be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:49:07.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.025+0000 7fb5e7dd4640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e0137c00 con 0x7fb5e00b4d60 2024-03-25T19:49:07.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.026+0000 7fb5e7dd4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e013bd60 con 0x7fb5e00b1730 2024-03-25T19:49:07.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.026+0000 7fb5e7dd4640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e0135db0 con 0x7fb5e014be90 2024-03-25T19:49:07.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.026+0000 7fb5e7dd4640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:07.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.026+0000 7fb5e7dd4640 10 monclient(hunting): authenticate will time out at 4382.107888s 2024-03-25T19:49:07.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.036+0000 7fb5e6371640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fb5e014be90 0x7fb5e01db530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:44402/0 (socket says 172.21.15.81:44402) 2024-03-25T19:49:07.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.036+0000 7fb5e6371640 1 -- 172.21.15.81:0/961790776 learned_addr learned my addr 172.21.15.81:0/961790776 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:49:07.038 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.038+0000 7fb5d67fc640 10 client.?.objecter ms_handle_connect 0x7fb5e014be90 2024-03-25T19:49:07.038 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.038+0000 7fb5d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:07.043 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.044+0000 7fb5d67fc640 10 client.?.objecter ms_handle_connect 0x7fb5e00b1730 2024-03-25T19:49:07.043 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.044+0000 7fb5d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:07.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.051+0000 7fb5d67fc640 10 client.?.objecter ms_handle_connect 0x7fb5e00b4d60 2024-03-25T19:49:07.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.051+0000 7fb5d67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:07.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.056+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934691752 0 0) 0x7fb5e0135db0 con 0x7fb5e014be90 2024-03-25T19:49:07.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.056+0000 7fb5d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:07.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.056+0000 7fb5d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:07.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.056+0000 7fb5d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:07.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.056+0000 7fb5d67fc640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:49:07.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.056+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5c4004eb0 con 0x7fb5e014be90 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.058+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628610814 0 0) 0x7fb5e013bd60 con 0x7fb5e00b1730 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.058+0000 7fb5d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.058+0000 7fb5d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.058+0000 7fb5d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.058+0000 7fb5d67fc640 10 monclient(hunting): my global_id is 4543 2024-03-25T19:49:07.058 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.058+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5e0135db0 con 0x7fb5e00b1730 2024-03-25T19:49:07.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.060+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829701416 0 0) 0x7fb5e0137c00 con 0x7fb5e00b4d60 2024-03-25T19:49:07.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.060+0000 7fb5d67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:07.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.060+0000 7fb5d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:07.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.060+0000 7fb5d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:07.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.060+0000 7fb5d67fc640 10 monclient(hunting): my global_id is 4557 2024-03-25T19:49:07.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.060+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5e013bd60 con 0x7fb5e00b4d60 2024-03-25T19:49:07.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.077+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1912403858 0 0) 0x7fb5c4004eb0 con 0x7fb5e014be90 2024-03-25T19:49:07.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.077+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5e0137c00 con 0x7fb5e014be90 2024-03-25T19:49:07.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.077+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb5d8001c30 con 0x7fb5e014be90 2024-03-25T19:49:07.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.077+0000 7fb5d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:07.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.077+0000 7fb5d67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:07.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.077+0000 7fb5d67fc640 10 monclient(hunting): dump: 2024-03-25T19:49:07.077 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:07.077 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:07.077 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:07.077 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:07.077 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:07.077 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:07.077 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:07.078 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:07.078 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:07.078 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:07.078 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.077+0000 7fb5d67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:07.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.082+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2463936234 0 0) 0x7fb5e013bd60 con 0x7fb5e00b4d60 2024-03-25T19:49:07.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.082+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5c4004eb0 con 0x7fb5e00b4d60 2024-03-25T19:49:07.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.082+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb5d0001b00 con 0x7fb5e00b4d60 2024-03-25T19:49:07.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.082+0000 7fb5d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:07.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.082+0000 7fb5d67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:07.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.082+0000 7fb5d67fc640 10 monclient(hunting): dump: 2024-03-25T19:49:07.082 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:07.082 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:07.082 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:07.082 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:07.083 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:07.083 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:07.083 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:07.083 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:07.083 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:07.083 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:07.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.091+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 482498030 0 0) 0x7fb5e0135db0 con 0x7fb5e00b1730 2024-03-25T19:49:07.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.091+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5e013bd60 con 0x7fb5e00b1730 2024-03-25T19:49:07.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.091+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb5dc0019f0 con 0x7fb5e00b1730 2024-03-25T19:49:07.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.091+0000 7fb5d67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:07.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.091+0000 7fb5d67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:07.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.091+0000 7fb5d67fc640 10 monclient(hunting): dump: 2024-03-25T19:49:07.092 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:07.092 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:07.092 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:07.092 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:07.092 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:07.092 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:07.092 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:07.092 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:07.092 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:07.092 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:07.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1489084607 0 0) 0x7fb5e0137c00 con 0x7fb5e014be90 2024-03-25T19:49:07.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 >> v1:172.21.15.98:6789/0 conn(0x7fb5e00b1730 legacy=0x7fb5e013b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:07.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 >> v1:172.21.15.81:6789/0 conn(0x7fb5e00b4d60 legacy=0x7fb5e01dae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:07.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5d67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:49:07.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5d67fc640 1 monclient: found mon.c 2024-03-25T19:49:07.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:07.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5d67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:49:07.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e0137e30 con 0x7fb5e014be90 2024-03-25T19:49:07.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5d67fc640 10 monclient: _finish_auth 0 2024-03-25T19:49:07.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5d67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:49:07.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:07.105 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5e7dd4640 5 monclient: authenticate success, global_id 4526 2024-03-25T19:49:07.105 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5e7dd4640 10 monclient: _renew_subs 2024-03-25T19:49:07.105 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5e7dd4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:49:07.105 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5e7dd4640 1 -- 172.21.15.81:0/961790776 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e01456b0 con 0x7fb5e014be90 2024-03-25T19:49:07.105 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5e7dd4640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:07.105 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5e7dd4640 10 monclient: _renew_subs 2024-03-25T19:49:07.105 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5e7dd4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:49:07.105 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5e7dd4640 1 -- 172.21.15.81:0/961790776 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e0136b60 con 0x7fb5e014be90 2024-03-25T19:49:07.105 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.104+0000 7fb5e7dd4640 1 librados: init done 2024-03-25T19:49:07.106 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.105+0000 7fb5e7dd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:07.106 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.105+0000 7fb5e7dd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:07.106 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.105+0000 7fb5e7dd4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:49:07.106 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.105+0000 7fb5e7dd4640 1 -- 172.21.15.81:0/961790776 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e0135c10 con 0x7fb5e014be90 2024-03-25T19:49:07.414 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.414+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5d8003990 con 0x7fb5e014be90 2024-03-25T19:49:07.414 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.414+0000 7fb5d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:07.414 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.414+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb5d8003c00 con 0x7fb5e014be90 2024-03-25T19:49:07.414 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.414+0000 7fb5d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:07.415 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.414+0000 7fb5d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:07.415 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.414+0000 7fb5d67fc640 10 monclient: dump: 2024-03-25T19:49:07.415 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:07.415 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:07.415 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:07.415 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:07.415 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:07.415 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:07.415 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:07.415 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:07.415 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:07.415 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:07.415 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.414+0000 7fb5d7fff640 4 set_mon_vals no callback set 2024-03-25T19:49:07.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.415+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fb5d8003e20 con 0x7fb5e014be90 2024-03-25T19:49:07.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.416+0000 7fb5e5b70640 10 monclient: get_auth_request con 0x7fb5c406e110 auth_method 0 2024-03-25T19:49:07.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.416+0000 7fb5d67fc640 10 client.4526.objecter ms_handle_connect 0x7fb5c406e110 2024-03-25T19:49:07.422 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.423+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fb5d8054800 con 0x7fb5e014be90 2024-03-25T19:49:07.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.423+0000 7fb5d67fc640 10 client.4526.objecter ms_dispatch 0x7fb5e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:07.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.423+0000 7fb5d67fc640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:07.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.423+0000 7fb5d67fc640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:07.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.423+0000 7fb5d67fc640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-25T19:49:07.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.423+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb5d8050310 con 0x7fb5e014be90 2024-03-25T19:49:07.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.423+0000 7fb5d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:07.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.423+0000 7fb5d67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:07.653 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.654+0000 7fb5e7dd4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:07.654 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.654+0000 7fb5e7dd4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:07.654 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.654+0000 7fb5e7dd4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:49:07.654 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.654+0000 7fb5e7dd4640 1 -- 172.21.15.81:0/961790776 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5e00627b0 con 0x7fb5e014be90 2024-03-25T19:49:07.692 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.692+0000 7fb5d67fc640 1 -- 172.21.15.81:0/961790776 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fb5d8052240 con 0x7fb5e014be90 2024-03-25T19:49:07.692 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.692+0000 7fb5d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:07.692 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.692+0000 7fb5d67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:07.693 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:49:07.693 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:07.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.695+0000 7fb5e7dd4640 10 librados: watch_flush enter 2024-03-25T19:49:07.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.695+0000 7fb5e7dd4640 10 librados: watch_flush exit 2024-03-25T19:49:07.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.695+0000 7fb5e7dd4640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-25T19:49:07.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.695+0000 7fb5e7dd4640 10 client.4526.objecter successfully canceled tick 2024-03-25T19:49:07.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.695+0000 7fb5e7dd4640 1 -- 172.21.15.81:0/961790776 >> v1:172.21.15.98:6800/3980829775 conn(0x7fb5c406e110 legacy=0x7fb5c40705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:07.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.695+0000 7fb5e7dd4640 10 monclient: shutdown 2024-03-25T19:49:07.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.695+0000 7fb5e7dd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:07.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.695+0000 7fb5e7dd4640 1 -- 172.21.15.81:0/961790776 >> v1:172.21.15.81:6790/0 conn(0x7fb5e014be90 legacy=0x7fb5e01db530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:07.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.695+0000 7fb5e7dd4640 1 -- 172.21.15.81:0/961790776 shutdown_connections 2024-03-25T19:49:07.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.695+0000 7fb5e7dd4640 1 -- 172.21.15.81:0/961790776 >> 172.21.15.81:0/961790776 conn(0x7fb5e00617a0 msgr2=0x7fb5e00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:07.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.695+0000 7fb5e7dd4640 1 -- 172.21.15.81:0/961790776 shutdown_connections 2024-03-25T19:49:07.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.695+0000 7fb5e7dd4640 1 -- 172.21.15.81:0/961790776 wait complete. 2024-03-25T19:49:07.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:07.695+0000 7fb5e7dd4640 1 librados: shutdown 2024-03-25T19:49:07.707 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:14.514 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:14.617 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.618+0000 7fb95e97c640 10 monclient: get_monmap_and_config 2024-03-25T19:49:14.618 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.618+0000 7fb95e97c640 10 monclient: build_initial_monmap 2024-03-25T19:49:14.618 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.618+0000 7fb95e97c640 10 monclient: monmap: 2024-03-25T19:49:14.618 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:49:14.618 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:14.618 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:49:14.619240+0000 2024-03-25T19:49:14.618 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:49:14.619240+0000 2024-03-25T19:49:14.618 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:14.618 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:14.618 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:49:14.618 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:49:14.618 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:49:14.618 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:14.619 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.619+0000 7fb95e97c640 1 Processor -- start 2024-03-25T19:49:14.619 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.619+0000 7fb95e97c640 1 -- start start 2024-03-25T19:49:14.619 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.619+0000 7fb95e97c640 10 monclient: init 2024-03-25T19:49:14.620 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.620+0000 7fb95e97c640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:14.620 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.620+0000 7fb95e97c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:14.620 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.620+0000 7fb95e97c640 10 monclient(hunting): picked mon.noname-a con 0x7fb9580b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:49:14.620 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.620+0000 7fb95e97c640 10 monclient(hunting): picked mon.noname-b con 0x7fb9580b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:49:14.620 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.620+0000 7fb95e97c640 10 monclient(hunting): picked mon.noname-c con 0x7fb95814be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:49:14.620 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.620+0000 7fb95e97c640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9580af940 con 0x7fb9580b4d60 2024-03-25T19:49:14.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.620+0000 7fb95e97c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9580b0a70 con 0x7fb95814be90 2024-03-25T19:49:14.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.620+0000 7fb95e97c640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9580b0d50 con 0x7fb9580b1730 2024-03-25T19:49:14.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.620+0000 7fb95e97c640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:14.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.620+0000 7fb95e97c640 10 monclient(hunting): authenticate will time out at 4389.702555s 2024-03-25T19:49:14.632 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.632+0000 7fb95cf19640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fb95814be90 0x7fb95814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:40484/0 (socket says 172.21.15.81:40484) 2024-03-25T19:49:14.632 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.632+0000 7fb95cf19640 1 -- 172.21.15.81:0/355670073 learned_addr learned my addr 172.21.15.81:0/355670073 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.641+0000 7fb956ffd640 1 -- 172.21.15.81:0/355670073 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723899053 0 0) 0x7fb9580b0a70 con 0x7fb95814be90 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.641+0000 7fb956ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.641+0000 7fb956ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.641+0000 7fb956ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.641+0000 7fb956ffd640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:49:14.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.641+0000 7fb956ffd640 1 -- 172.21.15.81:0/355670073 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb940003520 con 0x7fb95814be90 2024-03-25T19:49:14.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.647+0000 7fb956ffd640 1 -- 172.21.15.81:0/355670073 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3349141360 0 0) 0x7fb9580af940 con 0x7fb9580b4d60 2024-03-25T19:49:14.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.647+0000 7fb956ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:14.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.647+0000 7fb956ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:14.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.647+0000 7fb956ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:14.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.647+0000 7fb956ffd640 10 monclient(hunting): my global_id is 4560 2024-03-25T19:49:14.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.647+0000 7fb956ffd640 1 -- 172.21.15.81:0/355670073 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9580b0a70 con 0x7fb9580b4d60 2024-03-25T19:49:14.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.647+0000 7fb956ffd640 1 -- 172.21.15.81:0/355670073 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405623108 0 0) 0x7fb9580b0d50 con 0x7fb9580b1730 2024-03-25T19:49:14.648 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.647+0000 7fb956ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:14.648 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.647+0000 7fb956ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:14.648 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.647+0000 7fb956ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:14.648 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.647+0000 7fb956ffd640 10 monclient(hunting): my global_id is 4529 2024-03-25T19:49:14.648 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.647+0000 7fb956ffd640 1 -- 172.21.15.81:0/355670073 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9580af940 con 0x7fb9580b1730 2024-03-25T19:49:14.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.649+0000 7fb956ffd640 1 -- 172.21.15.81:0/355670073 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 643677698 0 0) 0x7fb940003520 con 0x7fb95814be90 2024-03-25T19:49:14.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.649+0000 7fb956ffd640 1 -- 172.21.15.81:0/355670073 >> v1:172.21.15.81:6790/0 conn(0x7fb9580b1730 legacy=0x7fb9580b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:14.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.649+0000 7fb956ffd640 1 -- 172.21.15.81:0/355670073 >> v1:172.21.15.81:6789/0 conn(0x7fb9580b4d60 legacy=0x7fb95813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:14.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.649+0000 7fb956ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:49:14.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.649+0000 7fb956ffd640 1 monclient: found mon.noname-c 2024-03-25T19:49:14.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.649+0000 7fb956ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:14.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.649+0000 7fb956ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:49:14.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.649+0000 7fb956ffd640 1 -- 172.21.15.81:0/355670073 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9580b1030 con 0x7fb95814be90 2024-03-25T19:49:14.649 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.649+0000 7fb956ffd640 10 monclient: _finish_auth 0 2024-03-25T19:49:14.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.649+0000 7fb956ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:49:14.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.649+0000 7fb956ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:14.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.649+0000 7fb956ffd640 1 -- 172.21.15.81:0/355670073 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb950001a10 con 0x7fb95814be90 2024-03-25T19:49:14.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.649+0000 7fb956ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:14.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.649+0000 7fb956ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:14.650 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.649+0000 7fb956ffd640 10 monclient: dump: 2024-03-25T19:49:14.650 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:14.650 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:14.650 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:14.650 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:14.650 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:14.650 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:14.650 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:14.650 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:14.651 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:14.651 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:14.651 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.649+0000 7fb95e97c640 5 monclient: authenticate success, global_id 4546 2024-03-25T19:49:14.651 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.649+0000 7fb95e97c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:14.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.659+0000 7fb956ffd640 1 -- 172.21.15.81:0/355670073 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb950001b70 con 0x7fb95814be90 2024-03-25T19:49:14.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.659+0000 7fb956ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:14.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.659+0000 7fb956ffd640 1 -- 172.21.15.81:0/355670073 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb950001f80 con 0x7fb95814be90 2024-03-25T19:49:14.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.659+0000 7fb95e97c640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:14.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.659+0000 7fb95e97c640 4 set_mon_vals no callback set 2024-03-25T19:49:14.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.659+0000 7fb956ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:14.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.659+0000 7fb956ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:14.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.659+0000 7fb956ffd640 10 monclient: dump: 2024-03-25T19:49:14.659 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:14.660 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:14.660 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:14.660 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:14.660 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:14.660 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:14.660 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:14.660 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:14.660 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:14.660 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:14.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.660+0000 7fb95e97c640 10 monclient: shutdown 2024-03-25T19:49:14.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.660+0000 7fb95e97c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:14.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.660+0000 7fb95e97c640 1 -- 172.21.15.81:0/355670073 >> v1:172.21.15.98:6789/0 conn(0x7fb95814be90 legacy=0x7fb95814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:14.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.660+0000 7fb95e97c640 1 -- 172.21.15.81:0/355670073 shutdown_connections 2024-03-25T19:49:14.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.660+0000 7fb95e97c640 1 -- 172.21.15.81:0/355670073 >> 172.21.15.81:0/355670073 conn(0x7fb9580617a0 msgr2=0x7fb958061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:14.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.660+0000 7fb95e97c640 1 -- 172.21.15.81:0/355670073 shutdown_connections 2024-03-25T19:49:14.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.660+0000 7fb95e97c640 1 -- 172.21.15.81:0/355670073 wait complete. 2024-03-25T19:49:14.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.661+0000 7fb95e97c640 10 monclient: build_initial_monmap 2024-03-25T19:49:14.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.661+0000 7fb95e97c640 10 monclient: monmap: 2024-03-25T19:49:14.662 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:49:14.662 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:14.662 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:49:14.662 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:49:14.662 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:14.662 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:49:14.662 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:49:14.662 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:49:14.663 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:49:14.663 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:14.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.661+0000 7fb95e97c640 1 librados: starting msgr at 2024-03-25T19:49:14.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.661+0000 7fb95e97c640 1 librados: starting objecter 2024-03-25T19:49:14.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.661+0000 7fb95e97c640 1 Processor -- start 2024-03-25T19:49:14.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.662+0000 7fb95e97c640 1 -- start start 2024-03-25T19:49:14.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.662+0000 7fb95e97c640 1 librados: setting wanted keys 2024-03-25T19:49:14.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.662+0000 7fb95e97c640 1 librados: calling monclient init 2024-03-25T19:49:14.663 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.662+0000 7fb95e97c640 10 monclient: init 2024-03-25T19:49:14.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.662+0000 7fb95e97c640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:14.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.662+0000 7fb95e97c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:14.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.662+0000 7fb95e97c640 10 monclient(hunting): picked mon.noname-a con 0x7fb9580b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:49:14.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.662+0000 7fb95e97c640 10 monclient(hunting): picked mon.noname-c con 0x7fb9580b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:49:14.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.662+0000 7fb95e97c640 10 monclient(hunting): picked mon.noname-b con 0x7fb95814be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:49:14.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.662+0000 7fb95e97c640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9581d75d0 con 0x7fb9580b1730 2024-03-25T19:49:14.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.662+0000 7fb95e97c640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb958141e10 con 0x7fb95814be90 2024-03-25T19:49:14.664 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.662+0000 7fb95e97c640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb958142080 con 0x7fb9580b4d60 2024-03-25T19:49:14.665 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.662+0000 7fb95e97c640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:14.665 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.662+0000 7fb95e97c640 10 monclient(hunting): authenticate will time out at 4389.744659s 2024-03-25T19:49:14.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.667+0000 7fb957fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fb9580b1730 0x7fb9581405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:43966/0 (socket says 172.21.15.81:43966) 2024-03-25T19:49:14.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.667+0000 7fb957fff640 1 -- 172.21.15.81:0/2799992454 learned_addr learned my addr 172.21.15.81:0/2799992454 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:49:14.668 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.668+0000 7fb954ff9640 10 client.?.objecter ms_handle_connect 0x7fb9580b1730 2024-03-25T19:49:14.668 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.668+0000 7fb954ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:14.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.673+0000 7fb954ff9640 10 client.?.objecter ms_handle_connect 0x7fb9580b4d60 2024-03-25T19:49:14.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.673+0000 7fb954ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:14.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.678+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175724402 0 0) 0x7fb9581d75d0 con 0x7fb9580b1730 2024-03-25T19:49:14.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.678+0000 7fb954ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:14.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.678+0000 7fb954ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:14.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.678+0000 7fb954ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:14.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.678+0000 7fb954ff9640 10 monclient(hunting): my global_id is 4563 2024-03-25T19:49:14.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.678+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb944004eb0 con 0x7fb9580b1730 2024-03-25T19:49:14.679 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.679+0000 7fb954ff9640 10 client.?.objecter ms_handle_connect 0x7fb95814be90 2024-03-25T19:49:14.679 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.679+0000 7fb954ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:14.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.684+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878075071 0 0) 0x7fb958142080 con 0x7fb9580b4d60 2024-03-25T19:49:14.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.684+0000 7fb954ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:14.684 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.684+0000 7fb954ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:14.684 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.684+0000 7fb954ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:14.684 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.684+0000 7fb954ff9640 10 monclient(hunting): my global_id is 4532 2024-03-25T19:49:14.684 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.684+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9581d75d0 con 0x7fb9580b4d60 2024-03-25T19:49:14.687 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.688+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385553298 0 0) 0x7fb958141e10 con 0x7fb95814be90 2024-03-25T19:49:14.687 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.688+0000 7fb954ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:14.687 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.688+0000 7fb954ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:14.687 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.688+0000 7fb954ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:14.687 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.688+0000 7fb954ff9640 10 monclient(hunting): my global_id is 4549 2024-03-25T19:49:14.687 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.688+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb958142080 con 0x7fb95814be90 2024-03-25T19:49:14.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.693+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3118027788 0 0) 0x7fb944004eb0 con 0x7fb9580b1730 2024-03-25T19:49:14.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.694+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb958141e10 con 0x7fb9580b1730 2024-03-25T19:49:14.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.694+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb948001a30 con 0x7fb9580b1730 2024-03-25T19:49:14.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.694+0000 7fb954ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:14.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.694+0000 7fb954ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:14.693 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.694+0000 7fb954ff9640 10 monclient(hunting): dump: 2024-03-25T19:49:14.694 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:14.694 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:14.694 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:14.694 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:14.694 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:14.694 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:14.694 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:14.694 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:14.694 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:14.694 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:14.694 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.694+0000 7fb954ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:14.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.695+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3957705797 0 0) 0x7fb9581d75d0 con 0x7fb9580b4d60 2024-03-25T19:49:14.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.695+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb944004eb0 con 0x7fb9580b4d60 2024-03-25T19:49:14.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.695+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb94c001870 con 0x7fb9580b4d60 2024-03-25T19:49:14.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.695+0000 7fb954ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:14.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.695+0000 7fb954ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:14.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.695+0000 7fb954ff9640 10 monclient(hunting): dump: 2024-03-25T19:49:14.695 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:14.696 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:14.696 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:14.696 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:14.696 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:14.696 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:14.696 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:14.696 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:14.696 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:14.696 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:14.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 562918514 0 0) 0x7fb944004eb0 con 0x7fb9580b4d60 2024-03-25T19:49:14.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 >> v1:172.21.15.98:6789/0 conn(0x7fb95814be90 legacy=0x7fb9581daf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:14.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 >> v1:172.21.15.81:6789/0 conn(0x7fb9580b1730 legacy=0x7fb9581405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:14.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb954ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:49:14.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb954ff9640 1 monclient: found mon.c 2024-03-25T19:49:14.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb954ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:14.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb954ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:49:14.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb958140d00 con 0x7fb9580b4d60 2024-03-25T19:49:14.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb954ff9640 10 monclient: _finish_auth 0 2024-03-25T19:49:14.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb954ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:49:14.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb954ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:14.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb95e97c640 5 monclient: authenticate success, global_id 4532 2024-03-25T19:49:14.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb95e97c640 10 monclient: _renew_subs 2024-03-25T19:49:14.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb95e97c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:49:14.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb95e97c640 1 -- 172.21.15.81:0/2799992454 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb958135a90 con 0x7fb9580b4d60 2024-03-25T19:49:14.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb95e97c640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:14.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb95e97c640 10 monclient: _renew_subs 2024-03-25T19:49:14.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb95e97c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:49:14.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb95e97c640 1 -- 172.21.15.81:0/2799992454 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9581d83a0 con 0x7fb9580b4d60 2024-03-25T19:49:14.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.704+0000 7fb95e97c640 1 librados: init done 2024-03-25T19:49:14.706 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.705+0000 7fb95e97c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:14.706 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.705+0000 7fb95e97c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:14.706 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.705+0000 7fb95e97c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:49:14.706 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:14.705+0000 7fb95e97c640 1 -- 172.21.15.81:0/2799992454 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9580b22c0 con 0x7fb9580b4d60 2024-03-25T19:49:15.015 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.016+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb94c003610 con 0x7fb9580b4d60 2024-03-25T19:49:15.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.016+0000 7fb954ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:15.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.016+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fb94c001d80 con 0x7fb9580b4d60 2024-03-25T19:49:15.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.016+0000 7fb954ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:15.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.016+0000 7fb954ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:15.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.016+0000 7fb954ff9640 10 monclient: dump: 2024-03-25T19:49:15.016 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:15.016 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:15.016 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:15.016 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:15.016 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:15.016 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:15.016 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:15.017 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:15.017 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:15.017 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:15.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.016+0000 7fb9567fc640 4 set_mon_vals no callback set 2024-03-25T19:49:15.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.016+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fb94c003b20 con 0x7fb9580b4d60 2024-03-25T19:49:15.018 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.017+0000 7fb957fff640 10 monclient: get_auth_request con 0x7fb94406dfb0 auth_method 0 2024-03-25T19:49:15.018 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.017+0000 7fb954ff9640 10 client.4532.objecter ms_handle_connect 0x7fb94406dfb0 2024-03-25T19:49:15.023 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.024+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fb94c0535c0 con 0x7fb9580b4d60 2024-03-25T19:49:15.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.024+0000 7fb954ff9640 10 client.4532.objecter ms_dispatch 0x7fb958000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:15.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.024+0000 7fb954ff9640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:15.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.024+0000 7fb954ff9640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:15.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.024+0000 7fb954ff9640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-25T19:49:15.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.024+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb94c04f0d0 con 0x7fb9580b4d60 2024-03-25T19:49:15.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.024+0000 7fb954ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:15.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.024+0000 7fb954ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:15.272 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.272+0000 7fb95e97c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:15.272 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.272+0000 7fb95e97c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:15.272 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.272+0000 7fb95e97c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:49:15.272 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.272+0000 7fb95e97c640 1 -- 172.21.15.81:0/2799992454 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9581dac30 con 0x7fb9580b4d60 2024-03-25T19:49:15.295 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.295+0000 7fb954ff9640 1 -- 172.21.15.81:0/2799992454 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fb9581dac30 con 0x7fb9580b4d60 2024-03-25T19:49:15.295 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.296+0000 7fb954ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:15.295 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.296+0000 7fb954ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:15.295 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:49:15.296 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:15.298 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.298+0000 7fb95e97c640 10 librados: watch_flush enter 2024-03-25T19:49:15.298 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.298+0000 7fb95e97c640 10 librados: watch_flush exit 2024-03-25T19:49:15.298 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.298+0000 7fb95e97c640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-25T19:49:15.299 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.298+0000 7fb95e97c640 10 client.4532.objecter successfully canceled tick 2024-03-25T19:49:15.299 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.298+0000 7fb95e97c640 1 -- 172.21.15.81:0/2799992454 >> v1:172.21.15.98:6800/3980829775 conn(0x7fb94406dfb0 legacy=0x7fb944070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:15.299 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.298+0000 7fb95e97c640 10 monclient: shutdown 2024-03-25T19:49:15.299 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.298+0000 7fb95e97c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:15.299 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.298+0000 7fb95e97c640 1 -- 172.21.15.81:0/2799992454 >> v1:172.21.15.81:6790/0 conn(0x7fb9580b4d60 legacy=0x7fb9581d7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:15.299 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.298+0000 7fb95e97c640 1 -- 172.21.15.81:0/2799992454 shutdown_connections 2024-03-25T19:49:15.299 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.298+0000 7fb95e97c640 1 -- 172.21.15.81:0/2799992454 >> 172.21.15.81:0/2799992454 conn(0x7fb9580617a0 msgr2=0x7fb9580b8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:15.299 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.298+0000 7fb95e97c640 1 -- 172.21.15.81:0/2799992454 shutdown_connections 2024-03-25T19:49:15.299 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.298+0000 7fb95e97c640 1 -- 172.21.15.81:0/2799992454 wait complete. 2024-03-25T19:49:15.299 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:15.298+0000 7fb95e97c640 1 librados: shutdown 2024-03-25T19:49:15.310 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:22.117 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:22.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.231+0000 7fd171a7b640 10 monclient: get_monmap_and_config 2024-03-25T19:49:22.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.231+0000 7fd171a7b640 10 monclient: build_initial_monmap 2024-03-25T19:49:22.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.231+0000 7fd171a7b640 10 monclient: monmap: 2024-03-25T19:49:22.231 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:49:22.231 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:22.231 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:49:22.232376+0000 2024-03-25T19:49:22.231 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:49:22.232376+0000 2024-03-25T19:49:22.232 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:22.232 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:22.232 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:49:22.232 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:49:22.232 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:49:22.232 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:22.233 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.232+0000 7fd171a7b640 1 Processor -- start 2024-03-25T19:49:22.233 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.232+0000 7fd171a7b640 1 -- start start 2024-03-25T19:49:22.233 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.232+0000 7fd171a7b640 10 monclient: init 2024-03-25T19:49:22.233 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.232+0000 7fd171a7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:22.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.232+0000 7fd171a7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:22.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.232+0000 7fd171a7b640 10 monclient(hunting): picked mon.noname-c con 0x7fd16c0b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:49:22.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.232+0000 7fd171a7b640 10 monclient(hunting): picked mon.noname-a con 0x7fd16c0b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:49:22.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.233+0000 7fd171a7b640 10 monclient(hunting): picked mon.noname-b con 0x7fd16c14be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:49:22.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.233+0000 7fd171a7b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c0af940 con 0x7fd16c0b1730 2024-03-25T19:49:22.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.233+0000 7fd171a7b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c0b0a70 con 0x7fd16c0b4d60 2024-03-25T19:49:22.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.233+0000 7fd171a7b640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c0b0d50 con 0x7fd16c14be90 2024-03-25T19:49:22.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.233+0000 7fd171a7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:22.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.233+0000 7fd171a7b640 10 monclient(hunting): authenticate will time out at 4397.314948s 2024-03-25T19:49:22.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.239+0000 7fd16bfff640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fd16c14be90 0x7fd16c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:42988/0 (socket says 172.21.15.81:42988) 2024-03-25T19:49:22.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.239+0000 7fd16bfff640 1 -- 172.21.15.81:0/93453605 learned_addr learned my addr 172.21.15.81:0/93453605 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:49:22.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.247+0000 7fd16a7fc640 1 -- 172.21.15.81:0/93453605 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961603074 0 0) 0x7fd16c0b0d50 con 0x7fd16c14be90 2024-03-25T19:49:22.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.248+0000 7fd16a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:22.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.248+0000 7fd16a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:22.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.248+0000 7fd16a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:22.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.248+0000 7fd16a7fc640 10 monclient(hunting): my global_id is 4535 2024-03-25T19:49:22.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.248+0000 7fd16a7fc640 1 -- 172.21.15.81:0/93453605 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd154003520 con 0x7fd16c14be90 2024-03-25T19:49:22.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.262+0000 7fd16a7fc640 1 -- 172.21.15.81:0/93453605 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564939057 0 0) 0x7fd16c0b0a70 con 0x7fd16c0b4d60 2024-03-25T19:49:22.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.262+0000 7fd16a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:22.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.262+0000 7fd16a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:22.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.262+0000 7fd16a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:22.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.262+0000 7fd16a7fc640 10 monclient(hunting): my global_id is 4552 2024-03-25T19:49:22.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.262+0000 7fd16a7fc640 1 -- 172.21.15.81:0/93453605 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd16c0b0d50 con 0x7fd16c0b4d60 2024-03-25T19:49:22.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.262+0000 7fd16a7fc640 1 -- 172.21.15.81:0/93453605 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144651717 0 0) 0x7fd16c0af940 con 0x7fd16c0b1730 2024-03-25T19:49:22.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.262+0000 7fd16a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:22.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.262+0000 7fd16a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:22.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.262+0000 7fd16a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:22.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.262+0000 7fd16a7fc640 10 monclient(hunting): my global_id is 4566 2024-03-25T19:49:22.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.262+0000 7fd16a7fc640 1 -- 172.21.15.81:0/93453605 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd16c0b0a70 con 0x7fd16c0b1730 2024-03-25T19:49:22.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.269+0000 7fd16a7fc640 1 -- 172.21.15.81:0/93453605 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3578498794 0 0) 0x7fd154003520 con 0x7fd16c14be90 2024-03-25T19:49:22.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.269+0000 7fd16a7fc640 1 -- 172.21.15.81:0/93453605 >> v1:172.21.15.98:6789/0 conn(0x7fd16c0b4d60 legacy=0x7fd16c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:22.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.269+0000 7fd16a7fc640 1 -- 172.21.15.81:0/93453605 >> v1:172.21.15.81:6789/0 conn(0x7fd16c0b1730 legacy=0x7fd16c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:22.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.269+0000 7fd16a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:49:22.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.269+0000 7fd16a7fc640 1 monclient: found mon.noname-b 2024-03-25T19:49:22.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.269+0000 7fd16a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:22.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.269+0000 7fd16a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:49:22.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.269+0000 7fd16a7fc640 1 -- 172.21.15.81:0/93453605 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c0b1030 con 0x7fd16c14be90 2024-03-25T19:49:22.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.269+0000 7fd16a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:49:22.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.269+0000 7fd16a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:49:22.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.269+0000 7fd16a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:22.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.269+0000 7fd16a7fc640 1 -- 172.21.15.81:0/93453605 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd164001a10 con 0x7fd16c14be90 2024-03-25T19:49:22.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.269+0000 7fd171a7b640 5 monclient: authenticate success, global_id 4535 2024-03-25T19:49:22.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.269+0000 7fd16a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:22.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.269+0000 7fd16a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:22.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.269+0000 7fd16a7fc640 10 monclient: dump: 2024-03-25T19:49:22.270 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:22.270 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:22.270 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:22.270 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:22.270 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:22.270 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:22.270 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:22.271 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:22.271 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:22.271 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:22.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.269+0000 7fd171a7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.281+0000 7fd16a7fc640 1 -- 172.21.15.81:0/93453605 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd164001b70 con 0x7fd16c14be90 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.281+0000 7fd16a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.281+0000 7fd16a7fc640 1 -- 172.21.15.81:0/93453605 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd164001f80 con 0x7fd16c14be90 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.281+0000 7fd16a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.281+0000 7fd16a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.281+0000 7fd16a7fc640 10 monclient: dump: 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:22.281 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:22.282 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:22.282 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:22.282 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:22.282 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:22.282 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:22.282 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:22.282 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:22.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.281+0000 7fd171a7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:22.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.281+0000 7fd171a7b640 4 set_mon_vals no callback set 2024-03-25T19:49:22.283 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.282+0000 7fd171a7b640 10 monclient: shutdown 2024-03-25T19:49:22.283 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.282+0000 7fd171a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:22.283 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.282+0000 7fd171a7b640 1 -- 172.21.15.81:0/93453605 >> v1:172.21.15.81:6790/0 conn(0x7fd16c14be90 legacy=0x7fd16c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:22.284 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.282+0000 7fd171a7b640 1 -- 172.21.15.81:0/93453605 shutdown_connections 2024-03-25T19:49:22.284 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.282+0000 7fd171a7b640 1 -- 172.21.15.81:0/93453605 >> 172.21.15.81:0/93453605 conn(0x7fd16c0617a0 msgr2=0x7fd16c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:22.284 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.282+0000 7fd171a7b640 1 -- 172.21.15.81:0/93453605 shutdown_connections 2024-03-25T19:49:22.284 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.282+0000 7fd171a7b640 1 -- 172.21.15.81:0/93453605 wait complete. 2024-03-25T19:49:22.284 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.283+0000 7fd171a7b640 10 monclient: build_initial_monmap 2024-03-25T19:49:22.284 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.283+0000 7fd171a7b640 10 monclient: monmap: 2024-03-25T19:49:22.284 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:49:22.284 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:22.284 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:49:22.284 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:49:22.284 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:22.284 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:49:22.284 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:49:22.284 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:49:22.284 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:49:22.285 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:22.285 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.283+0000 7fd171a7b640 1 librados: starting msgr at 2024-03-25T19:49:22.285 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.283+0000 7fd171a7b640 1 librados: starting objecter 2024-03-25T19:49:22.285 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.283+0000 7fd171a7b640 1 Processor -- start 2024-03-25T19:49:22.285 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.284+0000 7fd171a7b640 1 -- start start 2024-03-25T19:49:22.285 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.284+0000 7fd171a7b640 1 librados: setting wanted keys 2024-03-25T19:49:22.285 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.284+0000 7fd171a7b640 1 librados: calling monclient init 2024-03-25T19:49:22.285 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.284+0000 7fd171a7b640 10 monclient: init 2024-03-25T19:49:22.286 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.284+0000 7fd171a7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:22.286 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.284+0000 7fd171a7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:49:22.286 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.284+0000 7fd171a7b640 10 monclient(hunting): picked mon.noname-a con 0x7fd16c0b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:49:22.286 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.284+0000 7fd171a7b640 10 monclient(hunting): picked mon.noname-b con 0x7fd16c0b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:49:22.286 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.284+0000 7fd171a7b640 10 monclient(hunting): picked mon.noname-c con 0x7fd16c14be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:49:22.286 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.285+0000 7fd171a7b640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c1406f0 con 0x7fd16c0b1730 2024-03-25T19:49:22.286 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.285+0000 7fd171a7b640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c1dee70 con 0x7fd16c0b4d60 2024-03-25T19:49:22.286 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.285+0000 7fd171a7b640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16c1df850 con 0x7fd16c14be90 2024-03-25T19:49:22.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.285+0000 7fd171a7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:22.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.285+0000 7fd171a7b640 10 monclient(hunting): authenticate will time out at 4397.366976s 2024-03-25T19:49:22.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.288+0000 7fd16affd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd16c0b4d60 0x7fd16c1de050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:53800/0 (socket says 172.21.15.81:53800) 2024-03-25T19:49:22.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.288+0000 7fd16affd640 1 -- 172.21.15.81:0/4072753454 learned_addr learned my addr 172.21.15.81:0/4072753454 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:49:22.289 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.289+0000 7fd153fff640 10 client.?.objecter ms_handle_connect 0x7fd16c0b4d60 2024-03-25T19:49:22.289 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.289+0000 7fd153fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:22.291 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.292+0000 7fd153fff640 10 client.?.objecter ms_handle_connect 0x7fd16c14be90 2024-03-25T19:49:22.291 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.292+0000 7fd153fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:22.299 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.299+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155536013 0 0) 0x7fd16c1dee70 con 0x7fd16c0b4d60 2024-03-25T19:49:22.299 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.299+0000 7fd153fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:22.299 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.299+0000 7fd153fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:22.299 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.299+0000 7fd153fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:22.299 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.299+0000 7fd153fff640 10 monclient(hunting): my global_id is 4555 2024-03-25T19:49:22.299 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.299+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd158004eb0 con 0x7fd16c0b4d60 2024-03-25T19:49:22.306 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.306+0000 7fd153fff640 10 client.?.objecter ms_handle_connect 0x7fd16c0b1730 2024-03-25T19:49:22.306 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.306+0000 7fd153fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:22.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.308+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119387652 0 0) 0x7fd16c1df850 con 0x7fd16c14be90 2024-03-25T19:49:22.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.308+0000 7fd153fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:22.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.308+0000 7fd153fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:22.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.308+0000 7fd153fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:22.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.308+0000 7fd153fff640 10 monclient(hunting): my global_id is 4538 2024-03-25T19:49:22.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.309+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd16c1dee70 con 0x7fd16c14be90 2024-03-25T19:49:22.310 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.310+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1159109749 0 0) 0x7fd158004eb0 con 0x7fd16c0b4d60 2024-03-25T19:49:22.310 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.311+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd16c1df850 con 0x7fd16c0b4d60 2024-03-25T19:49:22.311 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.311+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd160001a10 con 0x7fd16c0b4d60 2024-03-25T19:49:22.311 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.311+0000 7fd153fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:22.311 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.311+0000 7fd153fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:22.311 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.311+0000 7fd153fff640 10 monclient(hunting): dump: 2024-03-25T19:49:22.311 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:22.311 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:22.311 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:22.311 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:22.311 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:22.311 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:22.311 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:22.311 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:22.311 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:22.311 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:22.312 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.311+0000 7fd153fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:22.315 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.316+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055255171 0 0) 0x7fd16c1406f0 con 0x7fd16c0b1730 2024-03-25T19:49:22.315 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.316+0000 7fd153fff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:22.315 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.316+0000 7fd153fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:22.315 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.316+0000 7fd153fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:22.316 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.316+0000 7fd153fff640 10 monclient(hunting): my global_id is 4569 2024-03-25T19:49:22.316 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.316+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd158004eb0 con 0x7fd16c0b1730 2024-03-25T19:49:22.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.317+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1357302478 0 0) 0x7fd16c1df850 con 0x7fd16c0b4d60 2024-03-25T19:49:22.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.317+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 >> v1:172.21.15.81:6790/0 conn(0x7fd16c14be90 legacy=0x7fd16c1de760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:22.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.317+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 >> v1:172.21.15.81:6789/0 conn(0x7fd16c0b1730 legacy=0x7fd16c1df140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:22.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.317+0000 7fd153fff640 10 monclient: _finish_hunting 0 2024-03-25T19:49:22.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.317+0000 7fd153fff640 1 monclient: found mon.b 2024-03-25T19:49:22.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.317+0000 7fd153fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:22.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.317+0000 7fd153fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:22.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.317+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c141700 con 0x7fd16c0b4d60 2024-03-25T19:49:22.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.317+0000 7fd153fff640 10 monclient: _finish_auth 0 2024-03-25T19:49:22.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.318+0000 7fd153fff640 10 monclient: _check_auth_tickets 2024-03-25T19:49:22.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.318+0000 7fd153fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:22.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.318+0000 7fd171a7b640 5 monclient: authenticate success, global_id 4555 2024-03-25T19:49:22.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.318+0000 7fd171a7b640 10 monclient: _renew_subs 2024-03-25T19:49:22.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.318+0000 7fd171a7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:22.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.318+0000 7fd171a7b640 1 -- 172.21.15.81:0/4072753454 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16c1419e0 con 0x7fd16c0b4d60 2024-03-25T19:49:22.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.318+0000 7fd171a7b640 10 client.4555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:22.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.318+0000 7fd171a7b640 10 monclient: _renew_subs 2024-03-25T19:49:22.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.318+0000 7fd171a7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:22.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.318+0000 7fd171a7b640 1 -- 172.21.15.81:0/4072753454 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16c140380 con 0x7fd16c0b4d60 2024-03-25T19:49:22.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.318+0000 7fd171a7b640 1 librados: init done 2024-03-25T19:49:22.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.319+0000 7fd171a7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:22.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.319+0000 7fd171a7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:22.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.319+0000 7fd171a7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:22.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.319+0000 7fd171a7b640 1 -- 172.21.15.81:0/4072753454 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16c135820 con 0x7fd16c0b4d60 2024-03-25T19:49:22.328 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.329+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd160001c70 con 0x7fd16c0b4d60 2024-03-25T19:49:22.328 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.329+0000 7fd153fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:22.328 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.329+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd160003a60 con 0x7fd16c0b4d60 2024-03-25T19:49:22.328 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.329+0000 7fd153fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:22.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.329+0000 7fd153fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:22.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.329+0000 7fd153fff640 10 monclient: dump: 2024-03-25T19:49:22.329 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:22.329 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:22.329 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:22.329 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:22.329 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:22.329 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:22.329 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:22.329 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:22.329 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:22.329 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:22.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.329+0000 7fd169ffb640 4 set_mon_vals no callback set 2024-03-25T19:49:22.594 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.595+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fd160003bc0 con 0x7fd16c0b4d60 2024-03-25T19:49:22.595 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.595+0000 7fd16b7fe640 10 monclient: get_auth_request con 0x7fd15806de20 auth_method 0 2024-03-25T19:49:22.595 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.596+0000 7fd153fff640 10 client.4555.objecter ms_handle_connect 0x7fd15806de20 2024-03-25T19:49:22.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.598+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fd160054870 con 0x7fd16c0b4d60 2024-03-25T19:49:22.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.598+0000 7fd153fff640 10 client.4555.objecter ms_dispatch 0x7fd16c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:22.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.598+0000 7fd153fff640 3 client.4555.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:22.598 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.598+0000 7fd153fff640 3 client.4555.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:22.598 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.598+0000 7fd153fff640 20 client.4555.objecter dump_active .. 0 homeless 2024-03-25T19:49:22.602 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.603+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd1600501b0 con 0x7fd16c0b4d60 2024-03-25T19:49:22.602 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.603+0000 7fd153fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:22.603 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.603+0000 7fd153fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:22.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.840+0000 7fd171a7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:22.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.840+0000 7fd171a7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:22.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.840+0000 7fd171a7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:22.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.840+0000 7fd171a7b640 1 -- 172.21.15.81:0/4072753454 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd16c1408a0 con 0x7fd16c0b4d60 2024-03-25T19:49:22.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.861+0000 7fd153fff640 1 -- 172.21.15.81:0/4072753454 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fd16c1408a0 con 0x7fd16c0b4d60 2024-03-25T19:49:22.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.861+0000 7fd153fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:22.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.861+0000 7fd153fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:22.861 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:49:22.861 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:22.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.863+0000 7fd171a7b640 10 librados: watch_flush enter 2024-03-25T19:49:22.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.863+0000 7fd171a7b640 10 librados: watch_flush exit 2024-03-25T19:49:22.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.863+0000 7fd171a7b640 20 client.4555.objecter shutdown clearing up homeless session... 2024-03-25T19:49:22.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.863+0000 7fd171a7b640 10 client.4555.objecter successfully canceled tick 2024-03-25T19:49:22.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.863+0000 7fd171a7b640 1 -- 172.21.15.81:0/4072753454 >> v1:172.21.15.98:6800/3980829775 conn(0x7fd15806de20 legacy=0x7fd1580702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:22.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.863+0000 7fd171a7b640 10 monclient: shutdown 2024-03-25T19:49:22.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.863+0000 7fd171a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:22.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.863+0000 7fd171a7b640 1 -- 172.21.15.81:0/4072753454 >> v1:172.21.15.98:6789/0 conn(0x7fd16c0b4d60 legacy=0x7fd16c1de050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:22.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.863+0000 7fd171a7b640 1 -- 172.21.15.81:0/4072753454 shutdown_connections 2024-03-25T19:49:22.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.863+0000 7fd171a7b640 1 -- 172.21.15.81:0/4072753454 >> 172.21.15.81:0/4072753454 conn(0x7fd16c0617a0 msgr2=0x7fd16c13c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:22.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.863+0000 7fd171a7b640 1 -- 172.21.15.81:0/4072753454 shutdown_connections 2024-03-25T19:49:22.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.863+0000 7fd171a7b640 1 -- 172.21.15.81:0/4072753454 wait complete. 2024-03-25T19:49:22.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:22.863+0000 7fd171a7b640 1 librados: shutdown 2024-03-25T19:49:22.874 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:22.945 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:07:45.587 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:22.952 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:07:45.593 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:22.976 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:07:45.617 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:23.003 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:45.637 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:23.003 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:07:45.645 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:23.007 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:45.641 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:23.008 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:07:45.650 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:23.037 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:07:45.679 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:23.037 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:07:45.672 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:29.680 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:29.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.800+0000 7f0b9d58d640 10 monclient: get_monmap_and_config 2024-03-25T19:49:29.801 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.800+0000 7f0b9d58d640 10 monclient: build_initial_monmap 2024-03-25T19:49:29.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.800+0000 7f0b9d58d640 10 monclient: monmap: 2024-03-25T19:49:29.802 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:49:29.802 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:29.802 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:49:29.802735+0000 2024-03-25T19:49:29.802 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:49:29.802735+0000 2024-03-25T19:49:29.802 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:29.802 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:29.802 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:49:29.802 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:49:29.802 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:49:29.802 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:29.803 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.801+0000 7f0b9d58d640 1 Processor -- start 2024-03-25T19:49:29.803 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.801+0000 7f0b9d58d640 1 -- start start 2024-03-25T19:49:29.804 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.801+0000 7f0b9d58d640 10 monclient: init 2024-03-25T19:49:29.804 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.802+0000 7f0b9d58d640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:29.804 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.802+0000 7f0b9d58d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:29.804 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.802+0000 7f0b9d58d640 10 monclient(hunting): picked mon.noname-c con 0x7f0b980a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:49:29.804 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.802+0000 7f0b9d58d640 10 monclient(hunting): picked mon.noname-a con 0x7f0b9813f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:49:29.804 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.802+0000 7f0b9d58d640 10 monclient(hunting): picked mon.noname-b con 0x7f0b981430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:49:29.804 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.802+0000 7f0b9d58d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b98142520 con 0x7f0b9813f710 2024-03-25T19:49:29.804 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.802+0000 7f0b9d58d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b980b28a0 con 0x7f0b980a98f0 2024-03-25T19:49:29.804 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.802+0000 7f0b9d58d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b980b2a30 con 0x7f0b981430c0 2024-03-25T19:49:29.804 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.802+0000 7f0b9d58d640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:29.804 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.802+0000 7f0b9d58d640 10 monclient(hunting): authenticate will time out at 4404.885436s 2024-03-25T19:49:29.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.823+0000 7f0b967fc640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f0b9813f710 0x7f0b981429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:42808/0 (socket says 172.21.15.81:42808) 2024-03-25T19:49:29.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.823+0000 7f0b967fc640 1 -- 172.21.15.81:0/283612567 learned_addr learned my addr 172.21.15.81:0/283612567 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:49:29.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.843+0000 7f0b95ffb640 1 -- 172.21.15.81:0/283612567 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165695409 0 0) 0x7f0b98142520 con 0x7f0b9813f710 2024-03-25T19:49:29.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.843+0000 7f0b95ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:29.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.843+0000 7f0b95ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:29.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.843+0000 7f0b95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:29.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.843+0000 7f0b95ffb640 10 monclient(hunting): my global_id is 4572 2024-03-25T19:49:29.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.843+0000 7f0b95ffb640 1 -- 172.21.15.81:0/283612567 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b80003520 con 0x7f0b9813f710 2024-03-25T19:49:29.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.844+0000 7f0b95ffb640 1 -- 172.21.15.81:0/283612567 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504699059 0 0) 0x7f0b980b2a30 con 0x7f0b981430c0 2024-03-25T19:49:29.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.844+0000 7f0b95ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:29.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.844+0000 7f0b95ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:29.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.844+0000 7f0b95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:29.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.844+0000 7f0b95ffb640 10 monclient(hunting): my global_id is 4541 2024-03-25T19:49:29.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.844+0000 7f0b95ffb640 1 -- 172.21.15.81:0/283612567 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b98142520 con 0x7f0b981430c0 2024-03-25T19:49:29.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.849+0000 7f0b95ffb640 1 -- 172.21.15.81:0/283612567 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475511733 0 0) 0x7f0b980b28a0 con 0x7f0b980a98f0 2024-03-25T19:49:29.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.849+0000 7f0b95ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:29.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.849+0000 7f0b95ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:29.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.849+0000 7f0b95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:29.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.849+0000 7f0b95ffb640 10 monclient(hunting): my global_id is 4558 2024-03-25T19:49:29.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.849+0000 7f0b95ffb640 1 -- 172.21.15.81:0/283612567 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b980b2a30 con 0x7f0b980a98f0 2024-03-25T19:49:29.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b95ffb640 1 -- 172.21.15.81:0/283612567 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 350259534 0 0) 0x7f0b98142520 con 0x7f0b981430c0 2024-03-25T19:49:29.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b95ffb640 1 -- 172.21.15.81:0/283612567 >> v1:172.21.15.98:6789/0 conn(0x7f0b980a98f0 legacy=0x7f0b980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:29.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b95ffb640 1 -- 172.21.15.81:0/283612567 >> v1:172.21.15.81:6789/0 conn(0x7f0b9813f710 legacy=0x7f0b981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:29.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b95ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:49:29.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b95ffb640 1 monclient: found mon.noname-b 2024-03-25T19:49:29.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b95ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:29.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b95ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:49:29.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b95ffb640 1 -- 172.21.15.81:0/283612567 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b980b44c0 con 0x7f0b981430c0 2024-03-25T19:49:29.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b95ffb640 10 monclient: _finish_auth 0 2024-03-25T19:49:29.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b95ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:49:29.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:29.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b95ffb640 1 -- 172.21.15.81:0/283612567 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f0b900019f0 con 0x7f0b981430c0 2024-03-25T19:49:29.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b9d58d640 5 monclient: authenticate success, global_id 4541 2024-03-25T19:49:29.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b9d58d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:29.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:29.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b95ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:29.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b95ffb640 10 monclient: dump: 2024-03-25T19:49:29.860 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:29.860 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:29.861 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:29.861 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:29.861 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:29.861 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:29.861 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:29.861 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:29.861 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:29.861 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:29.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.858+0000 7f0b9d58d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:29.882 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.881+0000 7f0b95ffb640 1 -- 172.21.15.81:0/283612567 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b90001b50 con 0x7f0b981430c0 2024-03-25T19:49:29.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.881+0000 7f0b95ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:29.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.881+0000 7f0b95ffb640 1 -- 172.21.15.81:0/283612567 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f0b90001f60 con 0x7f0b981430c0 2024-03-25T19:49:29.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.881+0000 7f0b95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:29.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.881+0000 7f0b95ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:29.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.881+0000 7f0b95ffb640 10 monclient: dump: 2024-03-25T19:49:29.883 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:29.883 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:29.883 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:29.883 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:29.883 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:29.883 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:29.883 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:29.883 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:29.883 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:29.884 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:29.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.881+0000 7f0b9d58d640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:29.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.881+0000 7f0b9d58d640 4 set_mon_vals no callback set 2024-03-25T19:49:29.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.881+0000 7f0b9d58d640 10 monclient: shutdown 2024-03-25T19:49:29.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.881+0000 7f0b9d58d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:29.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.881+0000 7f0b9d58d640 1 -- 172.21.15.81:0/283612567 >> v1:172.21.15.81:6790/0 conn(0x7f0b981430c0 legacy=0x7f0b9814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:29.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.882+0000 7f0b9d58d640 1 -- 172.21.15.81:0/283612567 shutdown_connections 2024-03-25T19:49:29.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.882+0000 7f0b9d58d640 1 -- 172.21.15.81:0/283612567 >> 172.21.15.81:0/283612567 conn(0x7f0b980b7ea0 msgr2=0x7f0b980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:29.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.882+0000 7f0b9d58d640 1 -- 172.21.15.81:0/283612567 shutdown_connections 2024-03-25T19:49:29.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.882+0000 7f0b9d58d640 1 -- 172.21.15.81:0/283612567 wait complete. 2024-03-25T19:49:29.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.883+0000 7f0b9d58d640 10 monclient: build_initial_monmap 2024-03-25T19:49:29.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.883+0000 7f0b9d58d640 10 monclient: monmap: 2024-03-25T19:49:29.886 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:49:29.886 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:29.886 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:49:29.886 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:49:29.886 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:29.886 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:49:29.886 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:49:29.886 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:49:29.886 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:49:29.887 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:29.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.883+0000 7f0b9d58d640 1 librados: starting msgr at 2024-03-25T19:49:29.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.883+0000 7f0b9d58d640 1 librados: starting objecter 2024-03-25T19:49:29.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.883+0000 7f0b9d58d640 1 Processor -- start 2024-03-25T19:49:29.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.883+0000 7f0b9d58d640 1 -- start start 2024-03-25T19:49:29.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.883+0000 7f0b9d58d640 1 librados: setting wanted keys 2024-03-25T19:49:29.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.883+0000 7f0b9d58d640 1 librados: calling monclient init 2024-03-25T19:49:29.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.883+0000 7f0b9d58d640 10 monclient: init 2024-03-25T19:49:29.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.884+0000 7f0b9d58d640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:29.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.884+0000 7f0b9d58d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:29.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.884+0000 7f0b9d58d640 10 monclient(hunting): picked mon.noname-a con 0x7f0b980a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:49:29.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.884+0000 7f0b9d58d640 10 monclient(hunting): picked mon.noname-c con 0x7f0b9813f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:49:29.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.884+0000 7f0b9d58d640 10 monclient(hunting): picked mon.noname-b con 0x7f0b981430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:49:29.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.884+0000 7f0b9d58d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b980abac0 con 0x7f0b980a98f0 2024-03-25T19:49:29.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.884+0000 7f0b9d58d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b981d4310 con 0x7f0b981430c0 2024-03-25T19:49:29.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.884+0000 7f0b9d58d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b98138f10 con 0x7f0b9813f710 2024-03-25T19:49:29.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.885+0000 7f0b9d58d640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:29.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.885+0000 7f0b9d58d640 10 monclient(hunting): authenticate will time out at 4404.967862s 2024-03-25T19:49:29.895 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.895+0000 7f0b977fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f0b981430c0 0x7f0b981d4830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:46978/0 (socket says 172.21.15.81:46978) 2024-03-25T19:49:29.896 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.895+0000 7f0b977fe640 1 -- 172.21.15.81:0/667846220 learned_addr learned my addr 172.21.15.81:0/667846220 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:49:29.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.898+0000 7f0b7f7fe640 10 client.?.objecter ms_handle_connect 0x7f0b981430c0 2024-03-25T19:49:29.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.898+0000 7f0b7f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:29.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.903+0000 7f0b7f7fe640 10 client.?.objecter ms_handle_connect 0x7f0b9813f710 2024-03-25T19:49:29.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.903+0000 7f0b7f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:29.913 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.912+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430141808 0 0) 0x7f0b981d4310 con 0x7f0b981430c0 2024-03-25T19:49:29.913 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.912+0000 7f0b7f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:29.913 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.912+0000 7f0b7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:29.913 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.912+0000 7f0b7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:29.913 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.912+0000 7f0b7f7fe640 10 monclient(hunting): my global_id is 4561 2024-03-25T19:49:29.913 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.912+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b84004eb0 con 0x7f0b981430c0 2024-03-25T19:49:29.914 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.914+0000 7f0b7f7fe640 10 client.?.objecter ms_handle_connect 0x7f0b980a98f0 2024-03-25T19:49:29.914 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.914+0000 7f0b7f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.915+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071798458 0 0) 0x7f0b98138f10 con 0x7f0b9813f710 2024-03-25T19:49:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.915+0000 7f0b7f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.915+0000 7f0b7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.915+0000 7f0b7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.915+0000 7f0b7f7fe640 10 monclient(hunting): my global_id is 4544 2024-03-25T19:49:29.915 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.915+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b981d4310 con 0x7f0b9813f710 2024-03-25T19:49:29.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.924+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3264978741 0 0) 0x7f0b84004eb0 con 0x7f0b981430c0 2024-03-25T19:49:29.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.924+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b98138f10 con 0x7f0b981430c0 2024-03-25T19:49:29.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.924+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f0b90001fa0 con 0x7f0b981430c0 2024-03-25T19:49:29.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.924+0000 7f0b7f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:29.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.924+0000 7f0b7f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:29.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.924+0000 7f0b7f7fe640 10 monclient(hunting): dump: 2024-03-25T19:49:29.926 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:29.926 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:29.926 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:29.926 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:29.926 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:29.926 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:29.926 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:29.926 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:29.927 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:29.927 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:29.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.924+0000 7f0b7f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:29.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.927+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3747480709 0 0) 0x7f0b981d4310 con 0x7f0b9813f710 2024-03-25T19:49:29.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.927+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b84004eb0 con 0x7f0b9813f710 2024-03-25T19:49:29.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.927+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f0b8c001950 con 0x7f0b9813f710 2024-03-25T19:49:29.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.927+0000 7f0b7f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.927+0000 7f0b7f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.927+0000 7f0b7f7fe640 10 monclient(hunting): dump: 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.927+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037687231 0 0) 0x7f0b980abac0 con 0x7f0b980a98f0 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.927+0000 7f0b7f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:29.929 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.927+0000 7f0b7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:29.930 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.927+0000 7f0b7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:29.930 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.927+0000 7f0b7f7fe640 10 monclient(hunting): my global_id is 4575 2024-03-25T19:49:29.930 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.927+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b981d4310 con 0x7f0b980a98f0 2024-03-25T19:49:29.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2078136925 0 0) 0x7f0b98138f10 con 0x7f0b981430c0 2024-03-25T19:49:29.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 >> v1:172.21.15.81:6790/0 conn(0x7f0b9813f710 legacy=0x7f0b98134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:29.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 >> v1:172.21.15.81:6789/0 conn(0x7f0b980a98f0 legacy=0x7f0b98138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:29.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b7f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:49:29.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b7f7fe640 1 monclient: found mon.b 2024-03-25T19:49:29.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:29.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b7f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:29.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b98135010 con 0x7f0b981430c0 2024-03-25T19:49:29.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b7f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:49:29.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b7f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:49:29.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:29.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b9d58d640 5 monclient: authenticate success, global_id 4561 2024-03-25T19:49:29.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b9d58d640 10 monclient: _renew_subs 2024-03-25T19:49:29.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b9d58d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:29.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b9d58d640 1 -- 172.21.15.81:0/667846220 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b98134790 con 0x7f0b981430c0 2024-03-25T19:49:29.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b9d58d640 10 client.4561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:29.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b9d58d640 10 monclient: _renew_subs 2024-03-25T19:49:29.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b9d58d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:29.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b9d58d640 1 -- 172.21.15.81:0/667846220 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b981d5120 con 0x7f0b981430c0 2024-03-25T19:49:29.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.930+0000 7f0b9d58d640 1 librados: init done 2024-03-25T19:49:29.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.931+0000 7f0b9d58d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:29.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.931+0000 7f0b9d58d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:29.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.931+0000 7f0b9d58d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:29.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:29.931+0000 7f0b9d58d640 1 -- 172.21.15.81:0/667846220 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b981d44a0 con 0x7f0b981430c0 2024-03-25T19:49:30.158 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.157+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b90003c20 con 0x7f0b981430c0 2024-03-25T19:49:30.158 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.157+0000 7f0b7f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:30.158 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.157+0000 7f0b957fa640 4 set_mon_vals no callback set 2024-03-25T19:49:30.198 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.198+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f0b90003f70 con 0x7f0b981430c0 2024-03-25T19:49:30.198 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.198+0000 7f0b7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:30.199 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.198+0000 7f0b7f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:30.199 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.198+0000 7f0b7f7fe640 10 monclient: dump: 2024-03-25T19:49:30.199 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:30.199 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:30.199 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:30.199 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:30.199 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:30.199 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:30.199 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:30.199 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:30.199 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:30.199 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:30.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.207+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f0b90004150 con 0x7f0b981430c0 2024-03-25T19:49:30.209 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.208+0000 7f0b96ffd640 10 monclient: get_auth_request con 0x7f0b8406df50 auth_method 0 2024-03-25T19:49:30.209 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.208+0000 7f0b7f7fe640 10 client.4561.objecter ms_handle_connect 0x7f0b8406df50 2024-03-25T19:49:30.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.210+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f0b90054bd0 con 0x7f0b981430c0 2024-03-25T19:49:30.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.210+0000 7f0b7f7fe640 10 client.4561.objecter ms_dispatch 0x7f0b98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:30.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.210+0000 7f0b7f7fe640 3 client.4561.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:30.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.210+0000 7f0b7f7fe640 3 client.4561.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:30.211 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.210+0000 7f0b7f7fe640 20 client.4561.objecter dump_active .. 0 homeless 2024-03-25T19:49:30.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.213+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b900506e0 con 0x7f0b981430c0 2024-03-25T19:49:30.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.213+0000 7f0b7f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:30.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.213+0000 7f0b7f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:30.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.464+0000 7f0b9d58d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:30.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.464+0000 7f0b9d58d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:30.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.464+0000 7f0b9d58d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:30.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.464+0000 7f0b9d58d640 1 -- 172.21.15.81:0/667846220 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b981351a0 con 0x7f0b981430c0 2024-03-25T19:49:30.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.496+0000 7f0b7f7fe640 1 -- 172.21.15.81:0/667846220 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f0b90052610 con 0x7f0b981430c0 2024-03-25T19:49:30.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.496+0000 7f0b7f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:30.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.496+0000 7f0b7f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:30.497 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:49:30.497 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:30.499 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.498+0000 7f0b9d58d640 10 librados: watch_flush enter 2024-03-25T19:49:30.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.498+0000 7f0b9d58d640 10 librados: watch_flush exit 2024-03-25T19:49:30.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.498+0000 7f0b9d58d640 20 client.4561.objecter shutdown clearing up homeless session... 2024-03-25T19:49:30.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.498+0000 7f0b9d58d640 10 client.4561.objecter successfully canceled tick 2024-03-25T19:49:30.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.498+0000 7f0b9d58d640 1 -- 172.21.15.81:0/667846220 >> v1:172.21.15.98:6800/3980829775 conn(0x7f0b8406df50 legacy=0x7f0b84070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:30.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.498+0000 7f0b9d58d640 10 monclient: shutdown 2024-03-25T19:49:30.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.498+0000 7f0b9d58d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:30.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.498+0000 7f0b9d58d640 1 -- 172.21.15.81:0/667846220 >> v1:172.21.15.98:6789/0 conn(0x7f0b981430c0 legacy=0x7f0b981d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:30.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.498+0000 7f0b9d58d640 1 -- 172.21.15.81:0/667846220 shutdown_connections 2024-03-25T19:49:30.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.498+0000 7f0b9d58d640 1 -- 172.21.15.81:0/667846220 >> 172.21.15.81:0/667846220 conn(0x7f0b980b7ea0 msgr2=0x7f0b98074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:30.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.498+0000 7f0b9d58d640 1 -- 172.21.15.81:0/667846220 shutdown_connections 2024-03-25T19:49:30.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.498+0000 7f0b9d58d640 1 -- 172.21.15.81:0/667846220 wait complete. 2024-03-25T19:49:30.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:30.498+0000 7f0b9d58d640 1 librados: shutdown 2024-03-25T19:49:30.512 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:37.318 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:37.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.431+0000 7f695c4ff640 10 monclient: get_monmap_and_config 2024-03-25T19:49:37.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.431+0000 7f695c4ff640 10 monclient: build_initial_monmap 2024-03-25T19:49:37.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.431+0000 7f695c4ff640 10 monclient: monmap: 2024-03-25T19:49:37.431 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:49:37.431 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:37.431 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:49:37.432508+0000 2024-03-25T19:49:37.431 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:49:37.432508+0000 2024-03-25T19:49:37.431 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:37.431 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:37.431 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:49:37.432 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:49:37.432 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:49:37.432 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:37.432 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.432+0000 7f695c4ff640 1 Processor -- start 2024-03-25T19:49:37.432 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.432+0000 7f695c4ff640 1 -- start start 2024-03-25T19:49:37.432 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.432+0000 7f695c4ff640 10 monclient: init 2024-03-25T19:49:37.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.433+0000 7f695c4ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:37.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.433+0000 7f695c4ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:37.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.433+0000 7f695c4ff640 10 monclient(hunting): picked mon.noname-a con 0x7f69540a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:49:37.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.433+0000 7f695c4ff640 10 monclient(hunting): picked mon.noname-b con 0x7f695413f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:49:37.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.433+0000 7f695c4ff640 10 monclient(hunting): picked mon.noname-c con 0x7f69541430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:49:37.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.433+0000 7f695c4ff640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6954142520 con 0x7f69540a98f0 2024-03-25T19:49:37.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.433+0000 7f695c4ff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69540b28a0 con 0x7f69541430c0 2024-03-25T19:49:37.433 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.433+0000 7f695c4ff640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69540b2a30 con 0x7f695413f710 2024-03-25T19:49:37.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.433+0000 7f695c4ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:37.434 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.433+0000 7f695c4ff640 10 monclient(hunting): authenticate will time out at 4412.515497s 2024-03-25T19:49:37.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.444+0000 7f695aa9c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f69541430c0 0x7f695414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:51862/0 (socket says 172.21.15.81:51862) 2024-03-25T19:49:37.443 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.444+0000 7f695aa9c640 1 -- 172.21.15.81:0/3103570246 learned_addr learned my addr 172.21.15.81:0/3103570246 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:49:37.455 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.456+0000 7f6959299640 1 -- 172.21.15.81:0/3103570246 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583461072 0 0) 0x7f69540b28a0 con 0x7f69541430c0 2024-03-25T19:49:37.455 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.456+0000 7f6959299640 10 monclient(hunting): _negotiate 2024-03-25T19:49:37.455 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.456+0000 7f6959299640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:37.455 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.456+0000 7f6959299640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:37.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.456+0000 7f6959299640 10 monclient(hunting): my global_id is 4564 2024-03-25T19:49:37.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.456+0000 7f6959299640 1 -- 172.21.15.81:0/3103570246 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6940003520 con 0x7f69541430c0 2024-03-25T19:49:37.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.463+0000 7f6959299640 1 -- 172.21.15.81:0/3103570246 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3993496617 0 0) 0x7f6940003520 con 0x7f69541430c0 2024-03-25T19:49:37.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.463+0000 7f6959299640 1 -- 172.21.15.81:0/3103570246 >> v1:172.21.15.81:6790/0 conn(0x7f695413f710 legacy=0x7f69541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:37.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.463+0000 7f6959299640 1 -- 172.21.15.81:0/3103570246 >> v1:172.21.15.81:6789/0 conn(0x7f69540a98f0 legacy=0x7f69540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:37.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.463+0000 7f6959299640 10 monclient: _finish_hunting 0 2024-03-25T19:49:37.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.463+0000 7f6959299640 1 monclient: found mon.noname-c 2024-03-25T19:49:37.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.463+0000 7f6959299640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:37.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.463+0000 7f6959299640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:49:37.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.463+0000 7f6959299640 1 -- 172.21.15.81:0/3103570246 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69540b44c0 con 0x7f69541430c0 2024-03-25T19:49:37.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.463+0000 7f6959299640 10 monclient: _finish_auth 0 2024-03-25T19:49:37.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.463+0000 7f6959299640 10 monclient: _check_auth_tickets 2024-03-25T19:49:37.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.463+0000 7f6959299640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:37.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.463+0000 7f6959299640 1 -- 172.21.15.81:0/3103570246 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f694c001a10 con 0x7f69541430c0 2024-03-25T19:49:37.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.463+0000 7f6959299640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:37.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.463+0000 7f6959299640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:37.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.463+0000 7f6959299640 10 monclient: dump: 2024-03-25T19:49:37.464 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:37.464 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:37.464 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:37.464 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:37.464 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:37.464 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:37.465 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:37.465 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:37.465 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:37.465 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:37.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.463+0000 7f695c4ff640 5 monclient: authenticate success, global_id 4564 2024-03-25T19:49:37.465 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.463+0000 7f695c4ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:37.470 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.470+0000 7f6959299640 1 -- 172.21.15.81:0/3103570246 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f694c001b70 con 0x7f69541430c0 2024-03-25T19:49:37.470 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.470+0000 7f6959299640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:37.470 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.470+0000 7f695c4ff640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:37.470 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.470+0000 7f695c4ff640 4 set_mon_vals no callback set 2024-03-25T19:49:37.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.471+0000 7f695c4ff640 10 monclient: shutdown 2024-03-25T19:49:37.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.471+0000 7f695c4ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:37.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.471+0000 7f695c4ff640 1 -- 172.21.15.81:0/3103570246 >> v1:172.21.15.98:6789/0 conn(0x7f69541430c0 legacy=0x7f695414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:37.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.471+0000 7f695c4ff640 1 -- 172.21.15.81:0/3103570246 shutdown_connections 2024-03-25T19:49:37.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.471+0000 7f695c4ff640 1 -- 172.21.15.81:0/3103570246 >> 172.21.15.81:0/3103570246 conn(0x7f69540b7ea0 msgr2=0x7f69540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:37.471 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.471+0000 7f695c4ff640 1 -- 172.21.15.81:0/3103570246 shutdown_connections 2024-03-25T19:49:37.472 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.471+0000 7f695c4ff640 1 -- 172.21.15.81:0/3103570246 wait complete. 2024-03-25T19:49:37.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.472+0000 7f695c4ff640 10 monclient: build_initial_monmap 2024-03-25T19:49:37.473 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.472+0000 7f695c4ff640 10 monclient: monmap: 2024-03-25T19:49:37.473 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:49:37.473 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:37.474 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:49:37.474 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:49:37.474 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:37.474 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:49:37.474 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:49:37.474 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:49:37.474 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:49:37.474 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:37.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.472+0000 7f695c4ff640 1 librados: starting msgr at 2024-03-25T19:49:37.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.472+0000 7f695c4ff640 1 librados: starting objecter 2024-03-25T19:49:37.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.472+0000 7f695c4ff640 1 Processor -- start 2024-03-25T19:49:37.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.473+0000 7f695c4ff640 1 -- start start 2024-03-25T19:49:37.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.473+0000 7f695c4ff640 1 librados: setting wanted keys 2024-03-25T19:49:37.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.473+0000 7f695c4ff640 1 librados: calling monclient init 2024-03-25T19:49:37.474 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.473+0000 7f695c4ff640 10 monclient: init 2024-03-25T19:49:37.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.473+0000 7f695c4ff640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:37.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.473+0000 7f695c4ff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:49:37.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.473+0000 7f695c4ff640 10 monclient(hunting): picked mon.noname-b con 0x7f69540a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:49:37.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.473+0000 7f695c4ff640 10 monclient(hunting): picked mon.noname-c con 0x7f695413f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:49:37.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.473+0000 7f695c4ff640 10 monclient(hunting): picked mon.noname-a con 0x7f69541430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:49:37.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.473+0000 7f695c4ff640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6954137fa0 con 0x7f69541430c0 2024-03-25T19:49:37.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.473+0000 7f695c4ff640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6954139580 con 0x7f69540a98f0 2024-03-25T19:49:37.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.473+0000 7f695c4ff640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69541379f0 con 0x7f695413f710 2024-03-25T19:49:37.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.473+0000 7f695c4ff640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:37.475 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.473+0000 7f695c4ff640 10 monclient(hunting): authenticate will time out at 4412.555442s 2024-03-25T19:49:37.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.482+0000 7f695a29b640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f69540a98f0 0x7f6954132f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:51884/0 (socket says 172.21.15.81:51884) 2024-03-25T19:49:37.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.482+0000 7f695a29b640 1 -- 172.21.15.81:0/750907817 learned_addr learned my addr 172.21.15.81:0/750907817 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:49:37.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.485+0000 7f694affd640 10 client.?.objecter ms_handle_connect 0x7f69540a98f0 2024-03-25T19:49:37.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.485+0000 7f694affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:37.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.494+0000 7f694affd640 10 client.?.objecter ms_handle_connect 0x7f69541430c0 2024-03-25T19:49:37.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.494+0000 7f694affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:37.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.496+0000 7f694affd640 1 -- 172.21.15.81:0/750907817 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540574031 0 0) 0x7f6954139580 con 0x7f69540a98f0 2024-03-25T19:49:37.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.496+0000 7f694affd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:37.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.496+0000 7f694affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:37.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.496+0000 7f694affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:37.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.496+0000 7f694affd640 10 monclient(hunting): my global_id is 4567 2024-03-25T19:49:37.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.496+0000 7f694affd640 1 -- 172.21.15.81:0/750907817 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6938004eb0 con 0x7f69540a98f0 2024-03-25T19:49:37.499 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.500+0000 7f694affd640 10 client.?.objecter ms_handle_connect 0x7f695413f710 2024-03-25T19:49:37.499 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.500+0000 7f694affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:37.502 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.502+0000 7f694affd640 1 -- 172.21.15.81:0/750907817 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2055558645 0 0) 0x7f6938004eb0 con 0x7f69540a98f0 2024-03-25T19:49:37.502 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.503+0000 7f694affd640 1 -- 172.21.15.81:0/750907817 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6954139580 con 0x7f69540a98f0 2024-03-25T19:49:37.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.503+0000 7f694affd640 1 -- 172.21.15.81:0/750907817 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f6950001a30 con 0x7f69540a98f0 2024-03-25T19:49:37.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.503+0000 7f694affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:37.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.503+0000 7f694affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:37.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.503+0000 7f694affd640 10 monclient(hunting): dump: 2024-03-25T19:49:37.503 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:37.503 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:37.503 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:37.503 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:37.503 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:37.503 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:37.503 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:37.503 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:37.504 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:37.504 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:37.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.503+0000 7f694affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:37.506 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.506+0000 7f694affd640 1 -- 172.21.15.81:0/750907817 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905872348 0 0) 0x7f6954137fa0 con 0x7f69541430c0 2024-03-25T19:49:37.506 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.506+0000 7f694affd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:37.506 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.506+0000 7f694affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:37.506 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.506+0000 7f694affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:37.506 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.506+0000 7f694affd640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:49:37.506 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.506+0000 7f694affd640 1 -- 172.21.15.81:0/750907817 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6938004eb0 con 0x7f69541430c0 2024-03-25T19:49:37.508 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f694affd640 1 -- 172.21.15.81:0/750907817 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3975887169 0 0) 0x7f6954139580 con 0x7f69540a98f0 2024-03-25T19:49:37.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f694affd640 1 -- 172.21.15.81:0/750907817 >> v1:172.21.15.81:6790/0 conn(0x7f695413f710 legacy=0x7f69541da990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:37.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f694affd640 1 -- 172.21.15.81:0/750907817 >> v1:172.21.15.81:6789/0 conn(0x7f69541430c0 legacy=0x7f69541e2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:37.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f694affd640 10 monclient: _finish_hunting 0 2024-03-25T19:49:37.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f694affd640 1 monclient: found mon.b 2024-03-25T19:49:37.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f694affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:37.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f694affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:37.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f694affd640 1 -- 172.21.15.81:0/750907817 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6954137b80 con 0x7f69540a98f0 2024-03-25T19:49:37.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f694affd640 10 monclient: _finish_auth 0 2024-03-25T19:49:37.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f694affd640 10 monclient: _check_auth_tickets 2024-03-25T19:49:37.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f694affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:37.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f695c4ff640 5 monclient: authenticate success, global_id 4567 2024-03-25T19:49:37.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f695c4ff640 10 monclient: _renew_subs 2024-03-25T19:49:37.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f695c4ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:37.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f695c4ff640 1 -- 172.21.15.81:0/750907817 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6954137390 con 0x7f69540a98f0 2024-03-25T19:49:37.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f695c4ff640 10 client.4567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:37.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f695c4ff640 10 monclient: _renew_subs 2024-03-25T19:49:37.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f695c4ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:37.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f695c4ff640 1 -- 172.21.15.81:0/750907817 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69540b78e0 con 0x7f69540a98f0 2024-03-25T19:49:37.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.508+0000 7f695c4ff640 1 librados: init done 2024-03-25T19:49:37.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.509+0000 7f695c4ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:37.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.509+0000 7f695c4ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:37.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.509+0000 7f695c4ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:37.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.509+0000 7f695c4ff640 1 -- 172.21.15.81:0/750907817 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6954137730 con 0x7f69540a98f0 2024-03-25T19:49:37.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.511+0000 7f694affd640 1 -- 172.21.15.81:0/750907817 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6950001cb0 con 0x7f69540a98f0 2024-03-25T19:49:37.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.511+0000 7f694affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:37.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.511+0000 7f6958a98640 4 set_mon_vals no callback set 2024-03-25T19:49:37.780 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.780+0000 7f694affd640 1 -- 172.21.15.81:0/750907817 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f69500038c0 con 0x7f69540a98f0 2024-03-25T19:49:37.780 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.780+0000 7f694affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:37.780 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.780+0000 7f694affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:37.780 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.780+0000 7f694affd640 10 monclient: dump: 2024-03-25T19:49:37.780 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:37.780 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:37.781 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:37.781 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:37.781 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:37.781 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:37.781 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:37.781 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:37.781 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:37.781 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:37.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.781+0000 7f694affd640 1 -- 172.21.15.81:0/750907817 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f6950003aa0 con 0x7f69540a98f0 2024-03-25T19:49:37.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.782+0000 7f6959a9a640 10 monclient: get_auth_request con 0x7f693806dd40 auth_method 0 2024-03-25T19:49:37.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.782+0000 7f694affd640 10 client.4567.objecter ms_handle_connect 0x7f693806dd40 2024-03-25T19:49:37.783 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.784+0000 7f694affd640 1 -- 172.21.15.81:0/750907817 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f6950004460 con 0x7f69540a98f0 2024-03-25T19:49:37.783 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.784+0000 7f694affd640 10 client.4567.objecter ms_dispatch 0x7f6954000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:37.783 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.784+0000 7f694affd640 3 client.4567.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:37.784 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.784+0000 7f694affd640 3 client.4567.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:37.784 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.784+0000 7f694affd640 20 client.4567.objecter dump_active .. 0 homeless 2024-03-25T19:49:37.785 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.786+0000 7f694affd640 1 -- 172.21.15.81:0/750907817 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6950050060 con 0x7f69540a98f0 2024-03-25T19:49:37.785 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.786+0000 7f694affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:37.785 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.786+0000 7f694affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:37.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.996+0000 7f695c4ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:37.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.996+0000 7f695c4ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:37.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.996+0000 7f695c4ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:37.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:37.996+0000 7f695c4ff640 1 -- 172.21.15.81:0/750907817 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69540627b0 con 0x7f69540a98f0 2024-03-25T19:49:38.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:38.035+0000 7f694affd640 1 -- 172.21.15.81:0/750907817 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f6950052160 con 0x7f69540a98f0 2024-03-25T19:49:38.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:38.035+0000 7f694affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:38.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:38.035+0000 7f694affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:38.035 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:49:38.035 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:38.038 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:38.038+0000 7f695c4ff640 10 librados: watch_flush enter 2024-03-25T19:49:38.038 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:38.038+0000 7f695c4ff640 10 librados: watch_flush exit 2024-03-25T19:49:38.038 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:38.038+0000 7f695c4ff640 20 client.4567.objecter shutdown clearing up homeless session... 2024-03-25T19:49:38.038 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:38.038+0000 7f695c4ff640 10 client.4567.objecter successfully canceled tick 2024-03-25T19:49:38.038 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:38.038+0000 7f695c4ff640 1 -- 172.21.15.81:0/750907817 >> v1:172.21.15.98:6800/3980829775 conn(0x7f693806dd40 legacy=0x7f6938070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:38.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:38.038+0000 7f695c4ff640 10 monclient: shutdown 2024-03-25T19:49:38.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:38.038+0000 7f695c4ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:38.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:38.038+0000 7f695c4ff640 1 -- 172.21.15.81:0/750907817 >> v1:172.21.15.98:6789/0 conn(0x7f69540a98f0 legacy=0x7f6954132f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:38.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:38.038+0000 7f695c4ff640 1 -- 172.21.15.81:0/750907817 shutdown_connections 2024-03-25T19:49:38.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:38.038+0000 7f695c4ff640 1 -- 172.21.15.81:0/750907817 >> 172.21.15.81:0/750907817 conn(0x7f69540b7ea0 msgr2=0x7f6954074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:38.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:38.038+0000 7f695c4ff640 1 -- 172.21.15.81:0/750907817 shutdown_connections 2024-03-25T19:49:38.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:38.038+0000 7f695c4ff640 1 -- 172.21.15.81:0/750907817 wait complete. 2024-03-25T19:49:38.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:38.038+0000 7f695c4ff640 1 librados: shutdown 2024-03-25T19:49:38.051 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:42.978 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:08:05.620 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:42.985 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:08:05.627 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:43.026 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:08:05.668 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:43.037 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:08:05.679 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:43.044 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:08:05.686 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:43.048 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:08:05.682 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:43.056 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:08:05.690 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:43.081 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:08:05.715 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:43.081 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:08:05.723 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:44.856 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:44.968 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.969+0000 7f1f0d367640 10 monclient: get_monmap_and_config 2024-03-25T19:49:44.968 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.969+0000 7f1f0d367640 10 monclient: build_initial_monmap 2024-03-25T19:49:44.968 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.969+0000 7f1f0d367640 10 monclient: monmap: 2024-03-25T19:49:44.968 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:49:44.969 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:44.969 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:49:44.970075+0000 2024-03-25T19:49:44.969 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:49:44.970075+0000 2024-03-25T19:49:44.969 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:44.969 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:44.969 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:49:44.969 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:49:44.969 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:49:44.969 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:44.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.969+0000 7f1f0d367640 1 Processor -- start 2024-03-25T19:49:44.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.969+0000 7f1f0d367640 1 -- start start 2024-03-25T19:49:44.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.969+0000 7f1f0d367640 10 monclient: init 2024-03-25T19:49:44.970 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.970+0000 7f1f0d367640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:44.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.970+0000 7f1f0d367640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:44.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.970+0000 7f1f0d367640 10 monclient(hunting): picked mon.noname-c con 0x7f1f080a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:49:44.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.970+0000 7f1f0d367640 10 monclient(hunting): picked mon.noname-a con 0x7f1f0813f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:49:44.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.970+0000 7f1f0d367640 10 monclient(hunting): picked mon.noname-b con 0x7f1f081430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:49:44.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.970+0000 7f1f0d367640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f08142520 con 0x7f1f0813f710 2024-03-25T19:49:44.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.970+0000 7f1f0d367640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f080b28a0 con 0x7f1f080a98f0 2024-03-25T19:49:44.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.970+0000 7f1f0d367640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f080b2a30 con 0x7f1f081430c0 2024-03-25T19:49:44.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.970+0000 7f1f0d367640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:44.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.970+0000 7f1f0d367640 10 monclient(hunting): authenticate will time out at 4420.052333s 2024-03-25T19:49:44.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.976+0000 7f1f077fe640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f1f081430c0 0x7f1f0814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:33616/0 (socket says 172.21.15.81:33616) 2024-03-25T19:49:44.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.976+0000 7f1f077fe640 1 -- 172.21.15.81:0/1756049148 learned_addr learned my addr 172.21.15.81:0/1756049148 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:49:44.980 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.980+0000 7f1f05ffb640 1 -- 172.21.15.81:0/1756049148 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502803970 0 0) 0x7f1f080b2a30 con 0x7f1f081430c0 2024-03-25T19:49:44.980 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.980+0000 7f1f05ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:44.980 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.980+0000 7f1f05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:44.980 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.980+0000 7f1f05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:44.980 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.980+0000 7f1f05ffb640 10 monclient(hunting): my global_id is 4553 2024-03-25T19:49:44.980 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.980+0000 7f1f05ffb640 1 -- 172.21.15.81:0/1756049148 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ef0003520 con 0x7f1f081430c0 2024-03-25T19:49:44.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f05ffb640 1 -- 172.21.15.81:0/1756049148 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 157529374 0 0) 0x7f1ef0003520 con 0x7f1f081430c0 2024-03-25T19:49:44.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f05ffb640 1 -- 172.21.15.81:0/1756049148 >> v1:172.21.15.98:6789/0 conn(0x7f1f080a98f0 legacy=0x7f1f080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:44.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f05ffb640 1 -- 172.21.15.81:0/1756049148 >> v1:172.21.15.81:6789/0 conn(0x7f1f0813f710 legacy=0x7f1f081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:49:44.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f05ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:49:44.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f05ffb640 1 monclient: found mon.noname-b 2024-03-25T19:49:44.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:44.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f05ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:49:44.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f05ffb640 1 -- 172.21.15.81:0/1756049148 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f080b44c0 con 0x7f1f081430c0 2024-03-25T19:49:44.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f05ffb640 10 monclient: _finish_auth 0 2024-03-25T19:49:44.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f05ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:49:44.996 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:44.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f05ffb640 1 -- 172.21.15.81:0/1756049148 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1f00001a10 con 0x7f1f081430c0 2024-03-25T19:49:44.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f0d367640 5 monclient: authenticate success, global_id 4553 2024-03-25T19:49:44.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f0d367640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:44.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:44.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f05ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:44.997 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f05ffb640 10 monclient: dump: 2024-03-25T19:49:44.997 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:44.997 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:44.997 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:44.997 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:44.997 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:44.997 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:44.997 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:44.997 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:44.997 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:44.997 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:44.998 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:44.996+0000 7f1f0d367640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:45.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.014+0000 7f1f05ffb640 1 -- 172.21.15.81:0/1756049148 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f00001be0 con 0x7f1f081430c0 2024-03-25T19:49:45.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.014+0000 7f1f05ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:45.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.014+0000 7f1f05ffb640 1 -- 172.21.15.81:0/1756049148 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1f00002030 con 0x7f1f081430c0 2024-03-25T19:49:45.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.014+0000 7f1f0d367640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:45.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.014+0000 7f1f0d367640 4 set_mon_vals no callback set 2024-03-25T19:49:45.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.014+0000 7f1f05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:45.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.014+0000 7f1f05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:45.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.014+0000 7f1f05ffb640 10 monclient: dump: 2024-03-25T19:49:45.014 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:45.014 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:45.014 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:45.014 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:45.014 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:45.014 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:45.014 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:45.014 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:45.015 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:45.015 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:45.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.014+0000 7f1f0d367640 10 monclient: shutdown 2024-03-25T19:49:45.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.014+0000 7f1f0d367640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:45.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.014+0000 7f1f0d367640 1 -- 172.21.15.81:0/1756049148 >> v1:172.21.15.81:6790/0 conn(0x7f1f081430c0 legacy=0x7f1f0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:45.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.015+0000 7f1f0d367640 1 -- 172.21.15.81:0/1756049148 shutdown_connections 2024-03-25T19:49:45.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.015+0000 7f1f0d367640 1 -- 172.21.15.81:0/1756049148 >> 172.21.15.81:0/1756049148 conn(0x7f1f080b7ea0 msgr2=0x7f1f080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:45.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.015+0000 7f1f0d367640 1 -- 172.21.15.81:0/1756049148 shutdown_connections 2024-03-25T19:49:45.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.015+0000 7f1f0d367640 1 -- 172.21.15.81:0/1756049148 wait complete. 2024-03-25T19:49:45.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.015+0000 7f1f0d367640 10 monclient: build_initial_monmap 2024-03-25T19:49:45.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.015+0000 7f1f0d367640 10 monclient: monmap: 2024-03-25T19:49:45.017 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:49:45.017 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:45.017 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:49:45.017 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:49:45.017 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:45.017 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:49:45.017 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:49:45.017 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:49:45.017 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:49:45.017 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:45.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.016+0000 7f1f0d367640 1 librados: starting msgr at 2024-03-25T19:49:45.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.016+0000 7f1f0d367640 1 librados: starting objecter 2024-03-25T19:49:45.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.016+0000 7f1f0d367640 1 Processor -- start 2024-03-25T19:49:45.018 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.016+0000 7f1f0d367640 1 -- start start 2024-03-25T19:49:45.018 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.016+0000 7f1f0d367640 1 librados: setting wanted keys 2024-03-25T19:49:45.018 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.016+0000 7f1f0d367640 1 librados: calling monclient init 2024-03-25T19:49:45.018 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.016+0000 7f1f0d367640 10 monclient: init 2024-03-25T19:49:45.019 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.017+0000 7f1f0d367640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:45.019 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.017+0000 7f1f0d367640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:45.019 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.017+0000 7f1f0d367640 10 monclient(hunting): picked mon.noname-a con 0x7f1f080a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:49:45.019 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.017+0000 7f1f0d367640 10 monclient(hunting): picked mon.noname-c con 0x7f1f0813f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:49:45.019 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.017+0000 7f1f0d367640 10 monclient(hunting): picked mon.noname-b con 0x7f1f081430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:49:45.019 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.017+0000 7f1f0d367640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f080abae0 con 0x7f1f080a98f0 2024-03-25T19:49:45.019 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.017+0000 7f1f0d367640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f081d4380 con 0x7f1f081430c0 2024-03-25T19:49:45.019 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.017+0000 7f1f0d367640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f08135220 con 0x7f1f0813f710 2024-03-25T19:49:45.019 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.017+0000 7f1f0d367640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:45.019 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.017+0000 7f1f0d367640 10 monclient(hunting): authenticate will time out at 4420.099496s 2024-03-25T19:49:45.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.028+0000 7f1f06ffd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f1f080a98f0 0x7f1f081d48a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:52584/0 (socket says 172.21.15.81:52584) 2024-03-25T19:49:45.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.028+0000 7f1f06ffd640 1 -- 172.21.15.81:0/1473987852 learned_addr learned my addr 172.21.15.81:0/1473987852 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:49:45.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.033+0000 7f1eef7fe640 10 client.?.objecter ms_handle_connect 0x7f1f081430c0 2024-03-25T19:49:45.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.033+0000 7f1eef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:45.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.033+0000 7f1eef7fe640 10 client.?.objecter ms_handle_connect 0x7f1f080a98f0 2024-03-25T19:49:45.033 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.033+0000 7f1eef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:45.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.035+0000 7f1eef7fe640 10 client.?.objecter ms_handle_connect 0x7f1f0813f710 2024-03-25T19:49:45.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.035+0000 7f1eef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:45.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.036+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492652359 0 0) 0x7f1f080abae0 con 0x7f1f080a98f0 2024-03-25T19:49:45.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.036+0000 7f1eef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:45.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.036+0000 7f1eef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:45.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.036+0000 7f1eef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:45.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.036+0000 7f1eef7fe640 10 monclient(hunting): my global_id is 4584 2024-03-25T19:49:45.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.036+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ef4004eb0 con 0x7f1f080a98f0 2024-03-25T19:49:45.041 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.042+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594295773 0 0) 0x7f1f081d4380 con 0x7f1f081430c0 2024-03-25T19:49:45.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.042+0000 7f1eef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:45.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.042+0000 7f1eef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:45.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.042+0000 7f1eef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:45.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.042+0000 7f1eef7fe640 10 monclient(hunting): my global_id is 4573 2024-03-25T19:49:45.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.042+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f080abae0 con 0x7f1f081430c0 2024-03-25T19:49:45.048 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.049+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 880214273 0 0) 0x7f1ef4004eb0 con 0x7f1f080a98f0 2024-03-25T19:49:45.049 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.049+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f081d4380 con 0x7f1f080a98f0 2024-03-25T19:49:45.049 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.049+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1ef8001a30 con 0x7f1f080a98f0 2024-03-25T19:49:45.049 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.049+0000 7f1eef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:45.049 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.049+0000 7f1eef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:45.049 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.049+0000 7f1eef7fe640 10 monclient(hunting): dump: 2024-03-25T19:49:45.049 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:45.049 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:45.049 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:45.049 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:45.049 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:45.049 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:45.049 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:45.050 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:45.050 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:45.050 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:45.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.049+0000 7f1eef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:45.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.049+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446795496 0 0) 0x7f1f08135220 con 0x7f1f0813f710 2024-03-25T19:49:45.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.049+0000 7f1eef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:45.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.049+0000 7f1eef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:45.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.049+0000 7f1eef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:45.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.049+0000 7f1eef7fe640 10 monclient(hunting): my global_id is 4556 2024-03-25T19:49:45.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.049+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ef4004eb0 con 0x7f1f0813f710 2024-03-25T19:49:45.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3983167249 0 0) 0x7f1f081d4380 con 0x7f1f080a98f0 2024-03-25T19:49:45.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 >> v1:172.21.15.81:6790/0 conn(0x7f1f0813f710 legacy=0x7f1f081383d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:45.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 >> v1:172.21.15.98:6789/0 conn(0x7f1f081430c0 legacy=0x7f1f081e20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:45.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1eef7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:49:45.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1eef7fe640 1 monclient: found mon.a 2024-03-25T19:49:45.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1eef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:45.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1eef7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:49:45.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f081353b0 con 0x7f1f080a98f0 2024-03-25T19:49:45.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1eef7fe640 10 monclient: _finish_auth 0 2024-03-25T19:49:45.055 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1eef7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:49:45.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1eef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:45.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1f0d367640 5 monclient: authenticate success, global_id 4584 2024-03-25T19:49:45.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1f0d367640 10 monclient: _renew_subs 2024-03-25T19:49:45.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1f0d367640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:49:45.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1f0d367640 1 -- 172.21.15.81:0/1473987852 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f081d4fb0 con 0x7f1f080a98f0 2024-03-25T19:49:45.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1f0d367640 10 client.4584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:45.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1f0d367640 10 monclient: _renew_subs 2024-03-25T19:49:45.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1f0d367640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:49:45.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1f0d367640 1 -- 172.21.15.81:0/1473987852 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f08138cc0 con 0x7f1f080a98f0 2024-03-25T19:49:45.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.055+0000 7f1f0d367640 1 librados: init done 2024-03-25T19:49:45.057 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.056+0000 7f1f0d367640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:45.057 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.056+0000 7f1f0d367640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:45.057 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.056+0000 7f1f0d367640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:49:45.057 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.056+0000 7f1f0d367640 1 -- 172.21.15.81:0/1473987852 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f080b3ef0 con 0x7f1f080a98f0 2024-03-25T19:49:45.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.369+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ef8001c70 con 0x7f1f080a98f0 2024-03-25T19:49:45.369 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.369+0000 7f1eef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:45.369 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.369+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1ef80038c0 con 0x7f1f080a98f0 2024-03-25T19:49:45.369 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.369+0000 7f1eef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:45.369 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.369+0000 7f1eef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:45.369 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.369+0000 7f1eef7fe640 10 monclient: dump: 2024-03-25T19:49:45.369 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:45.369 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:45.369 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:45.369 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:45.369 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:45.369 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:45.369 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:45.369 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:45.370 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:45.370 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:45.370 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.369+0000 7f1f057fa640 4 set_mon_vals no callback set 2024-03-25T19:49:45.370 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.369+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f1ef8003aa0 con 0x7f1f080a98f0 2024-03-25T19:49:45.370 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.370+0000 7f1f067fc640 10 monclient: get_auth_request con 0x7f1ef406de00 auth_method 0 2024-03-25T19:49:45.371 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.370+0000 7f1eef7fe640 10 client.4584.objecter ms_handle_connect 0x7f1ef406de00 2024-03-25T19:49:45.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.375+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f1ef8054710 con 0x7f1f080a98f0 2024-03-25T19:49:45.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.375+0000 7f1eef7fe640 10 client.4584.objecter ms_dispatch 0x7f1f08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:45.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.375+0000 7f1eef7fe640 3 client.4584.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:45.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.375+0000 7f1eef7fe640 3 client.4584.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:45.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.375+0000 7f1eef7fe640 20 client.4584.objecter dump_active .. 0 homeless 2024-03-25T19:49:45.376 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.375+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ef8050220 con 0x7f1f080a98f0 2024-03-25T19:49:45.376 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.376+0000 7f1eef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:45.376 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.376+0000 7f1eef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:45.603 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.603+0000 7f1f0d367640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:45.603 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.603+0000 7f1f0d367640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:45.603 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.603+0000 7f1f0d367640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:49:45.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.603+0000 7f1f0d367640 1 -- 172.21.15.81:0/1473987852 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f081d41e0 con 0x7f1f080a98f0 2024-03-25T19:49:45.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.624+0000 7f1eef7fe640 1 -- 172.21.15.81:0/1473987852 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f1f081d41e0 con 0x7f1f080a98f0 2024-03-25T19:49:45.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.624+0000 7f1eef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:45.624 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.624+0000 7f1eef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:45.624 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:49:45.624 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:45.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.626+0000 7f1f0d367640 10 librados: watch_flush enter 2024-03-25T19:49:45.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.626+0000 7f1f0d367640 10 librados: watch_flush exit 2024-03-25T19:49:45.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.626+0000 7f1f0d367640 20 client.4584.objecter shutdown clearing up homeless session... 2024-03-25T19:49:45.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.626+0000 7f1f0d367640 10 client.4584.objecter successfully canceled tick 2024-03-25T19:49:45.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.626+0000 7f1f0d367640 1 -- 172.21.15.81:0/1473987852 >> v1:172.21.15.98:6800/3980829775 conn(0x7f1ef406de00 legacy=0x7f1ef40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:45.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.626+0000 7f1f0d367640 10 monclient: shutdown 2024-03-25T19:49:45.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.626+0000 7f1f0d367640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:45.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.626+0000 7f1f0d367640 1 -- 172.21.15.81:0/1473987852 >> v1:172.21.15.81:6789/0 conn(0x7f1f080a98f0 legacy=0x7f1f081d48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:45.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.627+0000 7f1f0d367640 1 -- 172.21.15.81:0/1473987852 shutdown_connections 2024-03-25T19:49:45.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.627+0000 7f1f0d367640 1 -- 172.21.15.81:0/1473987852 >> 172.21.15.81:0/1473987852 conn(0x7f1f080b7ea0 msgr2=0x7f1f08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:45.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.627+0000 7f1f0d367640 1 -- 172.21.15.81:0/1473987852 shutdown_connections 2024-03-25T19:49:45.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.627+0000 7f1f0d367640 1 -- 172.21.15.81:0/1473987852 wait complete. 2024-03-25T19:49:45.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:45.627+0000 7f1f0d367640 1 librados: shutdown 2024-03-25T19:49:45.639 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:52.447 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:52.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.555+0000 7ff09a3e9640 10 monclient: get_monmap_and_config 2024-03-25T19:49:52.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.555+0000 7ff09a3e9640 10 monclient: build_initial_monmap 2024-03-25T19:49:52.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.555+0000 7ff09a3e9640 10 monclient: monmap: 2024-03-25T19:49:52.555 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:49:52.555 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:52.555 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:49:52.556511+0000 2024-03-25T19:49:52.555 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:49:52.556511+0000 2024-03-25T19:49:52.555 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:52.555 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:52.556 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:49:52.556 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:49:52.556 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:49:52.556 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:52.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.556+0000 7ff09a3e9640 1 Processor -- start 2024-03-25T19:49:52.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.556+0000 7ff09a3e9640 1 -- start start 2024-03-25T19:49:52.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.556+0000 7ff09a3e9640 10 monclient: init 2024-03-25T19:49:52.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.556+0000 7ff09a3e9640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:52.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.556+0000 7ff09a3e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:49:52.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.556+0000 7ff09a3e9640 10 monclient(hunting): picked mon.noname-b con 0x7ff0940b10d0 addr v1:172.21.15.81:6790/0 2024-03-25T19:49:52.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.556+0000 7ff09a3e9640 10 monclient(hunting): picked mon.noname-c con 0x7ff0940b4360 addr v1:172.21.15.98:6789/0 2024-03-25T19:49:52.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.556+0000 7ff09a3e9640 10 monclient(hunting): picked mon.noname-a con 0x7ff0940a9470 addr v1:172.21.15.81:6789/0 2024-03-25T19:49:52.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.556+0000 7ff09a3e9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0940a9ab0 con 0x7ff0940a9470 2024-03-25T19:49:52.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.556+0000 7ff09a3e9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0940b2400 con 0x7ff0940b4360 2024-03-25T19:49:52.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.556+0000 7ff09a3e9640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0940aa2a0 con 0x7ff0940b10d0 2024-03-25T19:49:52.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.556+0000 7ff09a3e9640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:52.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.556+0000 7ff09a3e9640 10 monclient(hunting): authenticate will time out at 4427.638799s 2024-03-25T19:49:52.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.557+0000 7ff0937fe640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff0940b4360 0x7ff0940b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:34864/0 (socket says 172.21.15.81:34864) 2024-03-25T19:49:52.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.557+0000 7ff0937fe640 1 -- 172.21.15.81:0/2373181955 learned_addr learned my addr 172.21.15.81:0/2373181955 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:49:52.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.597+0000 7ff092ffd640 1 -- 172.21.15.81:0/2373181955 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745063763 0 0) 0x7ff0940aa2a0 con 0x7ff0940b10d0 2024-03-25T19:49:52.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.597+0000 7ff092ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:52.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.597+0000 7ff092ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:52.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.597+0000 7ff092ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:52.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.597+0000 7ff092ffd640 10 monclient(hunting): my global_id is 4559 2024-03-25T19:49:52.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.597+0000 7ff092ffd640 1 -- 172.21.15.81:0/2373181955 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff07c003520 con 0x7ff0940b10d0 2024-03-25T19:49:52.613 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.613+0000 7ff092ffd640 1 -- 172.21.15.81:0/2373181955 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1654396505 0 0) 0x7ff07c003520 con 0x7ff0940b10d0 2024-03-25T19:49:52.613 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.613+0000 7ff092ffd640 1 -- 172.21.15.81:0/2373181955 >> v1:172.21.15.98:6789/0 conn(0x7ff0940b4360 legacy=0x7ff0940b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:52.613 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.613+0000 7ff092ffd640 1 -- 172.21.15.81:0/2373181955 >> v1:172.21.15.81:6789/0 conn(0x7ff0940a9470 legacy=0x7ff09414dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:52.613 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.613+0000 7ff092ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:49:52.613 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.613+0000 7ff092ffd640 1 monclient: found mon.noname-b 2024-03-25T19:49:52.613 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.613+0000 7ff092ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:52.613 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.613+0000 7ff092ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:49:52.613 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.613+0000 7ff092ffd640 1 -- 172.21.15.81:0/2373181955 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0940b14b0 con 0x7ff0940b10d0 2024-03-25T19:49:52.613 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.613+0000 7ff092ffd640 10 monclient: _finish_auth 0 2024-03-25T19:49:52.613 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.613+0000 7ff092ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:49:52.614 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.613+0000 7ff092ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:52.614 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.613+0000 7ff092ffd640 1 -- 172.21.15.81:0/2373181955 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7ff084001690 con 0x7ff0940b10d0 2024-03-25T19:49:52.614 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.613+0000 7ff092ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:52.614 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.613+0000 7ff092ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:52.614 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.613+0000 7ff092ffd640 10 monclient: dump: 2024-03-25T19:49:52.614 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:52.614 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:52.614 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:52.614 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:52.614 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:52.614 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:52.614 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:52.614 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:52.615 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:52.615 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:52.615 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.613+0000 7ff09a3e9640 5 monclient: authenticate success, global_id 4559 2024-03-25T19:49:52.615 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.613+0000 7ff09a3e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:52.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.630+0000 7ff092ffd640 1 -- 172.21.15.81:0/2373181955 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0840017f0 con 0x7ff0940b10d0 2024-03-25T19:49:52.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.630+0000 7ff092ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:52.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.630+0000 7ff092ffd640 1 -- 172.21.15.81:0/2373181955 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7ff084001c00 con 0x7ff0940b10d0 2024-03-25T19:49:52.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.630+0000 7ff09a3e9640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:52.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.630+0000 7ff09a3e9640 4 set_mon_vals no callback set 2024-03-25T19:49:52.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.630+0000 7ff092ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:52.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.630+0000 7ff092ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:52.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.630+0000 7ff092ffd640 10 monclient: dump: 2024-03-25T19:49:52.631 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:52.631 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:52.631 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:52.631 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:52.631 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:52.631 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:52.631 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:52.631 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:52.631 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:52.631 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:52.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.631+0000 7ff09a3e9640 10 monclient: shutdown 2024-03-25T19:49:52.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.631+0000 7ff09a3e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:52.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.631+0000 7ff09a3e9640 1 -- 172.21.15.81:0/2373181955 >> v1:172.21.15.81:6790/0 conn(0x7ff0940b10d0 legacy=0x7ff0940b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:52.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.631+0000 7ff09a3e9640 1 -- 172.21.15.81:0/2373181955 shutdown_connections 2024-03-25T19:49:52.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.631+0000 7ff09a3e9640 1 -- 172.21.15.81:0/2373181955 >> 172.21.15.81:0/2373181955 conn(0x7ff0940617a0 msgr2=0x7ff094061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:52.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.631+0000 7ff09a3e9640 1 -- 172.21.15.81:0/2373181955 shutdown_connections 2024-03-25T19:49:52.634 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.631+0000 7ff09a3e9640 1 -- 172.21.15.81:0/2373181955 wait complete. 2024-03-25T19:49:52.634 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.632+0000 7ff09a3e9640 10 monclient: build_initial_monmap 2024-03-25T19:49:52.634 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.632+0000 7ff09a3e9640 10 monclient: monmap: 2024-03-25T19:49:52.634 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:49:52.634 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:52.634 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:49:52.634 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:49:52.634 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:52.634 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:49:52.634 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:49:52.634 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:49:52.634 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:49:52.634 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:52.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.632+0000 7ff09a3e9640 1 librados: starting msgr at 2024-03-25T19:49:52.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.632+0000 7ff09a3e9640 1 librados: starting objecter 2024-03-25T19:49:52.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.632+0000 7ff09a3e9640 1 Processor -- start 2024-03-25T19:49:52.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.632+0000 7ff09a3e9640 1 -- start start 2024-03-25T19:49:52.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.632+0000 7ff09a3e9640 1 librados: setting wanted keys 2024-03-25T19:49:52.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.632+0000 7ff09a3e9640 1 librados: calling monclient init 2024-03-25T19:49:52.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.632+0000 7ff09a3e9640 10 monclient: init 2024-03-25T19:49:52.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.633+0000 7ff09a3e9640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:52.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.633+0000 7ff09a3e9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:52.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.633+0000 7ff09a3e9640 10 monclient(hunting): picked mon.noname-b con 0x7ff0940a9470 addr v1:172.21.15.98:6789/0 2024-03-25T19:49:52.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.633+0000 7ff09a3e9640 10 monclient(hunting): picked mon.noname-a con 0x7ff0940b10d0 addr v1:172.21.15.81:6789/0 2024-03-25T19:49:52.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.633+0000 7ff09a3e9640 10 monclient(hunting): picked mon.noname-c con 0x7ff0940b4360 addr v1:172.21.15.81:6790/0 2024-03-25T19:49:52.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.633+0000 7ff09a3e9640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0941db830 con 0x7ff0940b10d0 2024-03-25T19:49:52.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.633+0000 7ff09a3e9640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0940b0200 con 0x7ff0940a9470 2024-03-25T19:49:52.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.633+0000 7ff09a3e9640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0940b0390 con 0x7ff0940b4360 2024-03-25T19:49:52.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.633+0000 7ff09a3e9640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:52.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.633+0000 7ff09a3e9640 10 monclient(hunting): authenticate will time out at 4427.715284s 2024-03-25T19:49:52.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.643+0000 7ff098986640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7ff0940b4360 0x7ff09413e7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:49938/0 (socket says 172.21.15.81:49938) 2024-03-25T19:49:52.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.644+0000 7ff098986640 1 -- 172.21.15.81:0/3153360439 learned_addr learned my addr 172.21.15.81:0/3153360439 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:49:52.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.645+0000 7ff090ff9640 10 client.?.objecter ms_handle_connect 0x7ff0940b4360 2024-03-25T19:49:52.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.645+0000 7ff090ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:52.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.646+0000 7ff090ff9640 10 client.?.objecter ms_handle_connect 0x7ff0940b10d0 2024-03-25T19:49:52.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.646+0000 7ff090ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:52.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.660+0000 7ff090ff9640 10 client.?.objecter ms_handle_connect 0x7ff0940a9470 2024-03-25T19:49:52.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.660+0000 7ff090ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:52.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.669+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680906167 0 0) 0x7ff0941db830 con 0x7ff0940b10d0 2024-03-25T19:49:52.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.669+0000 7ff090ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:52.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.669+0000 7ff090ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:52.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.669+0000 7ff090ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:52.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.669+0000 7ff090ff9640 10 monclient(hunting): my global_id is 4590 2024-03-25T19:49:52.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.669+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff080004eb0 con 0x7ff0940b10d0 2024-03-25T19:49:52.671 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.671+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685485051 0 0) 0x7ff0940b0200 con 0x7ff0940a9470 2024-03-25T19:49:52.671 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.672+0000 7ff090ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:52.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.672+0000 7ff090ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:52.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.672+0000 7ff090ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:52.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.672+0000 7ff090ff9640 10 monclient(hunting): my global_id is 4579 2024-03-25T19:49:52.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.672+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0941db830 con 0x7ff0940a9470 2024-03-25T19:49:52.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.672+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 449499534 0 0) 0x7ff0940b0390 con 0x7ff0940b4360 2024-03-25T19:49:52.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.672+0000 7ff090ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:52.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.672+0000 7ff090ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:52.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.672+0000 7ff090ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:52.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.672+0000 7ff090ff9640 10 monclient(hunting): my global_id is 4562 2024-03-25T19:49:52.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.672+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0940b0200 con 0x7ff0940b4360 2024-03-25T19:49:52.684 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.685+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2561771329 0 0) 0x7ff0941db830 con 0x7ff0940a9470 2024-03-25T19:49:52.684 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.685+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0940b0390 con 0x7ff0940a9470 2024-03-25T19:49:52.684 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.685+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7ff084001b40 con 0x7ff0940a9470 2024-03-25T19:49:52.685 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.685+0000 7ff090ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:52.685 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.685+0000 7ff090ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:52.685 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.685+0000 7ff090ff9640 10 monclient(hunting): dump: 2024-03-25T19:49:52.685 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:52.685 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:52.685 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:52.685 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:52.685 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:52.685 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:52.685 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:52.685 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:52.685 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:52.685 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:52.685 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.685+0000 7ff090ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:52.688 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.688+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 461456567 0 0) 0x7ff080004eb0 con 0x7ff0940b10d0 2024-03-25T19:49:52.688 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.688+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0941db830 con 0x7ff0940b10d0 2024-03-25T19:49:52.688 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.688+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7ff088001800 con 0x7ff0940b10d0 2024-03-25T19:49:52.688 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.688+0000 7ff090ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:52.688 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.688+0000 7ff090ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:52.689 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.688+0000 7ff090ff9640 10 monclient(hunting): dump: 2024-03-25T19:49:52.689 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:52.689 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:52.689 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:52.689 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:52.689 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:52.689 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:52.689 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:52.689 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:52.689 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:52.689 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:52.694 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1100207505 0 0) 0x7ff0940b0390 con 0x7ff0940a9470 2024-03-25T19:49:52.694 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 >> v1:172.21.15.81:6790/0 conn(0x7ff0940b4360 legacy=0x7ff09413e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:52.694 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 >> v1:172.21.15.81:6789/0 conn(0x7ff0940b10d0 legacy=0x7ff0940aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:52.694 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff090ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:49:52.694 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff090ff9640 1 monclient: found mon.b 2024-03-25T19:49:52.694 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff090ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:52.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff090ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:52.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0941db310 con 0x7ff0940a9470 2024-03-25T19:49:52.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff090ff9640 10 monclient: _finish_auth 0 2024-03-25T19:49:52.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff090ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:49:52.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff090ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:52.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff09a3e9640 5 monclient: authenticate success, global_id 4579 2024-03-25T19:49:52.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff09a3e9640 10 monclient: _renew_subs 2024-03-25T19:49:52.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff09a3e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:52.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff09a3e9640 1 -- 172.21.15.81:0/3153360439 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0941db500 con 0x7ff0940a9470 2024-03-25T19:49:52.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff09a3e9640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:52.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff09a3e9640 10 monclient: _renew_subs 2024-03-25T19:49:52.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff09a3e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:52.695 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff09a3e9640 1 -- 172.21.15.81:0/3153360439 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0940af890 con 0x7ff0940a9470 2024-03-25T19:49:52.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.694+0000 7ff09a3e9640 1 librados: init done 2024-03-25T19:49:52.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.695+0000 7ff09a3e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:52.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.695+0000 7ff09a3e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:52.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.695+0000 7ff09a3e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:52.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.695+0000 7ff09a3e9640 1 -- 172.21.15.81:0/3153360439 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0940ac8a0 con 0x7ff0940a9470 2024-03-25T19:49:52.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.701+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0840038a0 con 0x7ff0940a9470 2024-03-25T19:49:52.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.701+0000 7ff090ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:52.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.701+0000 7ff0927fc640 4 set_mon_vals no callback set 2024-03-25T19:49:52.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.703+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7ff084003bb0 con 0x7ff0940a9470 2024-03-25T19:49:52.702 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.703+0000 7ff090ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:52.703 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.703+0000 7ff090ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:52.703 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.703+0000 7ff090ff9640 10 monclient: dump: 2024-03-25T19:49:52.703 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:49:52.703 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:49:52.703 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:52.703 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:49:52.703 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:52.703 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:49:52.703 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:49:52.703 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:49:52.703 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:49:52.703 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:49:52.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.971+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7ff084019b80 con 0x7ff0940a9470 2024-03-25T19:49:52.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.972+0000 7ff0937fe640 10 monclient: get_auth_request con 0x7ff08006df70 auth_method 0 2024-03-25T19:49:52.971 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.972+0000 7ff090ff9640 10 client.4579.objecter ms_handle_connect 0x7ff08006df70 2024-03-25T19:49:52.974 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.974+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7ff0840536e0 con 0x7ff0940a9470 2024-03-25T19:49:52.974 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.974+0000 7ff090ff9640 10 client.4579.objecter ms_dispatch 0x7ff094000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:52.974 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.974+0000 7ff090ff9640 3 client.4579.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:52.974 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.974+0000 7ff090ff9640 3 client.4579.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:52.974 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.974+0000 7ff090ff9640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-25T19:49:52.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.976+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff08404f1f0 con 0x7ff0940a9470 2024-03-25T19:49:52.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.976+0000 7ff090ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:52.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:52.976+0000 7ff090ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:53.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.221+0000 7ff09a3e9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:53.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.221+0000 7ff09a3e9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:53.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.221+0000 7ff09a3e9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:49:53.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.221+0000 7ff09a3e9640 1 -- 172.21.15.81:0/3153360439 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0940627b0 con 0x7ff0940a9470 2024-03-25T19:49:53.243 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.243+0000 7ff090ff9640 1 -- 172.21.15.81:0/3153360439 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7ff084051120 con 0x7ff0940a9470 2024-03-25T19:49:53.243 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.243+0000 7ff090ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:53.243 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.243+0000 7ff090ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:53.243 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:49:53.243 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:53.245 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.245+0000 7ff09a3e9640 10 librados: watch_flush enter 2024-03-25T19:49:53.245 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.245+0000 7ff09a3e9640 10 librados: watch_flush exit 2024-03-25T19:49:53.245 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.245+0000 7ff09a3e9640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-25T19:49:53.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.245+0000 7ff09a3e9640 10 client.4579.objecter successfully canceled tick 2024-03-25T19:49:53.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.245+0000 7ff09a3e9640 1 -- 172.21.15.81:0/3153360439 >> v1:172.21.15.98:6800/3980829775 conn(0x7ff08006df70 legacy=0x7ff080070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:53.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.245+0000 7ff09a3e9640 10 monclient: shutdown 2024-03-25T19:49:53.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.245+0000 7ff09a3e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:53.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.245+0000 7ff09a3e9640 1 -- 172.21.15.81:0/3153360439 >> v1:172.21.15.98:6789/0 conn(0x7ff0940a9470 legacy=0x7ff0941dac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:53.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.245+0000 7ff09a3e9640 1 -- 172.21.15.81:0/3153360439 shutdown_connections 2024-03-25T19:49:53.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.245+0000 7ff09a3e9640 1 -- 172.21.15.81:0/3153360439 >> 172.21.15.81:0/3153360439 conn(0x7ff0940617a0 msgr2=0x7ff094061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:53.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.245+0000 7ff09a3e9640 1 -- 172.21.15.81:0/3153360439 shutdown_connections 2024-03-25T19:49:53.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.245+0000 7ff09a3e9640 1 -- 172.21.15.81:0/3153360439 wait complete. 2024-03-25T19:49:53.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:49:53.246+0000 7ff09a3e9640 1 librados: shutdown 2024-03-25T19:49:53.256 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:00.063 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:00.175 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.175+0000 7fcc2cb16640 10 monclient: get_monmap_and_config 2024-03-25T19:50:00.175 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.175+0000 7fcc2cb16640 10 monclient: build_initial_monmap 2024-03-25T19:50:00.175 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.175+0000 7fcc2cb16640 10 monclient: monmap: 2024-03-25T19:50:00.175 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:50:00.175 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:00.175 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:50:00.176577+0000 2024-03-25T19:50:00.176 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:50:00.176577+0000 2024-03-25T19:50:00.176 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:00.176 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:00.176 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:50:00.176 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:50:00.176 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:50:00.176 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:00.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.176+0000 7fcc2cb16640 1 Processor -- start 2024-03-25T19:50:00.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.176+0000 7fcc2cb16640 1 -- start start 2024-03-25T19:50:00.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.176+0000 7fcc2cb16640 10 monclient: init 2024-03-25T19:50:00.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.177+0000 7fcc2cb16640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:00.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.177+0000 7fcc2cb16640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:50:00.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.177+0000 7fcc2cb16640 10 monclient(hunting): picked mon.noname-b con 0x7fcc280b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:50:00.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.177+0000 7fcc2cb16640 10 monclient(hunting): picked mon.noname-c con 0x7fcc280b1730 addr v1:172.21.15.98:6789/0 2024-03-25T19:50:00.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.177+0000 7fcc2cb16640 10 monclient(hunting): picked mon.noname-a con 0x7fcc2814be90 addr v1:172.21.15.81:6789/0 2024-03-25T19:50:00.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.177+0000 7fcc2cb16640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc280af940 con 0x7fcc2814be90 2024-03-25T19:50:00.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.177+0000 7fcc2cb16640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc280b0a70 con 0x7fcc280b1730 2024-03-25T19:50:00.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.177+0000 7fcc2cb16640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc280b0d50 con 0x7fcc280b4d60 2024-03-25T19:50:00.178 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.177+0000 7fcc2cb16640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:00.178 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.177+0000 7fcc2cb16640 10 monclient(hunting): authenticate will time out at 4435.259265s 2024-03-25T19:50:00.185 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.185+0000 7fcc26d9d640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fcc2814be90 0x7fcc2814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:51118/0 (socket says 172.21.15.81:51118) 2024-03-25T19:50:00.185 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.185+0000 7fcc26d9d640 1 -- 172.21.15.81:0/1193504780 learned_addr learned my addr 172.21.15.81:0/1193504780 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:50:00.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.203+0000 7fcc2559a640 1 -- 172.21.15.81:0/1193504780 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247922391 0 0) 0x7fcc280b0a70 con 0x7fcc280b1730 2024-03-25T19:50:00.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.203+0000 7fcc2559a640 10 monclient(hunting): _negotiate 2024-03-25T19:50:00.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.203+0000 7fcc2559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:00.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.203+0000 7fcc2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:00.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.203+0000 7fcc2559a640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:50:00.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.203+0000 7fcc2559a640 1 -- 172.21.15.81:0/1193504780 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc10003520 con 0x7fcc280b1730 2024-03-25T19:50:00.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.222+0000 7fcc2559a640 1 -- 172.21.15.81:0/1193504780 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472025644 0 0) 0x7fcc280af940 con 0x7fcc2814be90 2024-03-25T19:50:00.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.222+0000 7fcc2559a640 10 monclient(hunting): _negotiate 2024-03-25T19:50:00.221 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.222+0000 7fcc2559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:00.222 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.222+0000 7fcc2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:00.222 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.222+0000 7fcc2559a640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:50:00.222 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.222+0000 7fcc2559a640 1 -- 172.21.15.81:0/1193504780 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc280b0a70 con 0x7fcc2814be90 2024-03-25T19:50:00.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.231+0000 7fcc2559a640 1 -- 172.21.15.81:0/1193504780 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1163375735 0 0) 0x7fcc10003520 con 0x7fcc280b1730 2024-03-25T19:50:00.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.231+0000 7fcc2559a640 1 -- 172.21.15.81:0/1193504780 >> v1:172.21.15.81:6790/0 conn(0x7fcc280b4d60 legacy=0x7fcc2813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:00.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.231+0000 7fcc2559a640 1 -- 172.21.15.81:0/1193504780 >> v1:172.21.15.81:6789/0 conn(0x7fcc2814be90 legacy=0x7fcc2814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:00.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.232+0000 7fcc2559a640 10 monclient: _finish_hunting 0 2024-03-25T19:50:00.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.232+0000 7fcc2559a640 1 monclient: found mon.noname-c 2024-03-25T19:50:00.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.232+0000 7fcc2559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:00.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.232+0000 7fcc2559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:50:00.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.232+0000 7fcc2559a640 1 -- 172.21.15.81:0/1193504780 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc280b1030 con 0x7fcc280b1730 2024-03-25T19:50:00.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.232+0000 7fcc2559a640 10 monclient: _finish_auth 0 2024-03-25T19:50:00.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.232+0000 7fcc2559a640 10 monclient: _check_auth_tickets 2024-03-25T19:50:00.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.232+0000 7fcc2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:00.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.232+0000 7fcc2559a640 1 -- 172.21.15.81:0/1193504780 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fcc1c001690 con 0x7fcc280b1730 2024-03-25T19:50:00.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.232+0000 7fcc2cb16640 5 monclient: authenticate success, global_id 4582 2024-03-25T19:50:00.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.232+0000 7fcc2cb16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:00.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.232+0000 7fcc2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:00.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.232+0000 7fcc2559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:00.233 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.232+0000 7fcc2559a640 10 monclient: dump: 2024-03-25T19:50:00.233 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:00.233 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:00.233 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:00.233 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:00.233 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:00.233 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:00.233 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:00.233 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:00.233 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:00.233 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:00.233 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.232+0000 7fcc2cb16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:00.257 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.257+0000 7fcc2559a640 1 -- 172.21.15.81:0/1193504780 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc1c0017f0 con 0x7fcc280b1730 2024-03-25T19:50:00.257 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.257+0000 7fcc2559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:00.257 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.257+0000 7fcc2559a640 1 -- 172.21.15.81:0/1193504780 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fcc1c001c00 con 0x7fcc280b1730 2024-03-25T19:50:00.257 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.257+0000 7fcc2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:00.257 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.257+0000 7fcc2559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:00.257 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.257+0000 7fcc2559a640 10 monclient: dump: 2024-03-25T19:50:00.257 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:00.257 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:00.257 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:00.258 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:00.258 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:00.258 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:00.258 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:00.258 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:00.258 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:00.258 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:00.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.257+0000 7fcc2cb16640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:00.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.257+0000 7fcc2cb16640 4 set_mon_vals no callback set 2024-03-25T19:50:00.260 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.258+0000 7fcc2cb16640 10 monclient: shutdown 2024-03-25T19:50:00.260 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.258+0000 7fcc2cb16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:00.260 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.258+0000 7fcc2cb16640 1 -- 172.21.15.81:0/1193504780 >> v1:172.21.15.98:6789/0 conn(0x7fcc280b1730 legacy=0x7fcc280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:00.260 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.258+0000 7fcc2cb16640 1 -- 172.21.15.81:0/1193504780 shutdown_connections 2024-03-25T19:50:00.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.258+0000 7fcc2cb16640 1 -- 172.21.15.81:0/1193504780 >> 172.21.15.81:0/1193504780 conn(0x7fcc280617a0 msgr2=0x7fcc28061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:00.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.258+0000 7fcc2cb16640 1 -- 172.21.15.81:0/1193504780 shutdown_connections 2024-03-25T19:50:00.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.258+0000 7fcc2cb16640 1 -- 172.21.15.81:0/1193504780 wait complete. 2024-03-25T19:50:00.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.259+0000 7fcc2cb16640 10 monclient: build_initial_monmap 2024-03-25T19:50:00.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.259+0000 7fcc2cb16640 10 monclient: monmap: 2024-03-25T19:50:00.261 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:50:00.261 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:00.261 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:50:00.261 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:50:00.261 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:00.261 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:50:00.261 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:50:00.261 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:50:00.261 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:50:00.261 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:00.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.259+0000 7fcc2cb16640 1 librados: starting msgr at 2024-03-25T19:50:00.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.259+0000 7fcc2cb16640 1 librados: starting objecter 2024-03-25T19:50:00.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.259+0000 7fcc2cb16640 1 Processor -- start 2024-03-25T19:50:00.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.259+0000 7fcc2cb16640 1 -- start start 2024-03-25T19:50:00.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.259+0000 7fcc2cb16640 1 librados: setting wanted keys 2024-03-25T19:50:00.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.259+0000 7fcc2cb16640 1 librados: calling monclient init 2024-03-25T19:50:00.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.259+0000 7fcc2cb16640 10 monclient: init 2024-03-25T19:50:00.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.260+0000 7fcc2cb16640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:00.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.260+0000 7fcc2cb16640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:00.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.260+0000 7fcc2cb16640 10 monclient(hunting): picked mon.noname-c con 0x7fcc280b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:50:00.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.260+0000 7fcc2cb16640 10 monclient(hunting): picked mon.noname-a con 0x7fcc280b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:50:00.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.260+0000 7fcc2cb16640 10 monclient(hunting): picked mon.noname-b con 0x7fcc2814be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:50:00.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.260+0000 7fcc2cb16640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2813bfb0 con 0x7fcc280b4d60 2024-03-25T19:50:00.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.260+0000 7fcc2cb16640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc281458e0 con 0x7fcc2814be90 2024-03-25T19:50:00.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.260+0000 7fcc2cb16640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2812f7b0 con 0x7fcc280b1730 2024-03-25T19:50:00.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.260+0000 7fcc2cb16640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:00.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.260+0000 7fcc2cb16640 10 monclient(hunting): authenticate will time out at 4435.342656s 2024-03-25T19:50:00.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.265+0000 7fcc2659c640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fcc280b1730 0x7fcc2813b8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:44522/0 (socket says 172.21.15.81:44522) 2024-03-25T19:50:00.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.266+0000 7fcc2659c640 1 -- 172.21.15.81:0/118510269 learned_addr learned my addr 172.21.15.81:0/118510269 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:50:00.265 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.266+0000 7fcc16ffd640 10 client.?.objecter ms_handle_connect 0x7fcc280b1730 2024-03-25T19:50:00.266 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.266+0000 7fcc16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:00.276 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.277+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799885151 0 0) 0x7fcc2812f7b0 con 0x7fcc280b1730 2024-03-25T19:50:00.276 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.277+0000 7fcc16ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:00.276 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.277+0000 7fcc16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:00.276 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.277+0000 7fcc16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:00.277 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.277+0000 7fcc16ffd640 10 monclient(hunting): my global_id is 4568 2024-03-25T19:50:00.277 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.277+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc08004eb0 con 0x7fcc280b1730 2024-03-25T19:50:00.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.282+0000 7fcc16ffd640 10 client.?.objecter ms_handle_connect 0x7fcc280b4d60 2024-03-25T19:50:00.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.282+0000 7fcc16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:00.283 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.284+0000 7fcc16ffd640 10 client.?.objecter ms_handle_connect 0x7fcc2814be90 2024-03-25T19:50:00.283 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.284+0000 7fcc16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:00.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.293+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367364803 0 0) 0x7fcc2813bfb0 con 0x7fcc280b4d60 2024-03-25T19:50:00.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.293+0000 7fcc16ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:00.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.293+0000 7fcc16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:00.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.293+0000 7fcc16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:00.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.293+0000 7fcc16ffd640 10 monclient(hunting): my global_id is 4596 2024-03-25T19:50:00.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.293+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc2812f7b0 con 0x7fcc280b4d60 2024-03-25T19:50:00.294 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.294+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878813209 0 0) 0x7fcc281458e0 con 0x7fcc2814be90 2024-03-25T19:50:00.294 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.294+0000 7fcc16ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:00.294 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.294+0000 7fcc16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:00.294 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.294+0000 7fcc16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:00.294 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.294+0000 7fcc16ffd640 10 monclient(hunting): my global_id is 4585 2024-03-25T19:50:00.294 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.294+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc2813bfb0 con 0x7fcc2814be90 2024-03-25T19:50:00.295 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.296+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3488006794 0 0) 0x7fcc08004eb0 con 0x7fcc280b1730 2024-03-25T19:50:00.296 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.296+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc281458e0 con 0x7fcc280b1730 2024-03-25T19:50:00.296 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.296+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fcc18001a30 con 0x7fcc280b1730 2024-03-25T19:50:00.296 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.296+0000 7fcc16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:00.296 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.296+0000 7fcc16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:00.296 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.296+0000 7fcc16ffd640 10 monclient(hunting): dump: 2024-03-25T19:50:00.296 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:00.296 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:00.296 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:00.296 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:00.296 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:00.296 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:00.296 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:00.296 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:00.296 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:00.296 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:00.297 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.296+0000 7fcc16ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:00.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.321+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3274253291 0 0) 0x7fcc2812f7b0 con 0x7fcc280b4d60 2024-03-25T19:50:00.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.321+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc08004eb0 con 0x7fcc280b4d60 2024-03-25T19:50:00.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.321+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fcc1c001b00 con 0x7fcc280b4d60 2024-03-25T19:50:00.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.321+0000 7fcc16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:00.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.321+0000 7fcc16ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:00.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.321+0000 7fcc16ffd640 10 monclient(hunting): dump: 2024-03-25T19:50:00.322 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:00.322 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:00.322 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:00.322 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:00.322 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:00.322 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:00.322 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:00.322 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:00.322 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:00.322 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:00.325 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 974353124 0 0) 0x7fcc281458e0 con 0x7fcc280b1730 2024-03-25T19:50:00.325 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 >> v1:172.21.15.98:6789/0 conn(0x7fcc2814be90 legacy=0x7fcc2812f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:00.325 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 >> v1:172.21.15.81:6789/0 conn(0x7fcc280b4d60 legacy=0x7fcc2812e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:00.325 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc16ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:00.325 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc16ffd640 1 monclient: found mon.c 2024-03-25T19:50:00.325 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:00.325 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc16ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:50:00.325 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2813aa90 con 0x7fcc280b1730 2024-03-25T19:50:00.326 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc16ffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:00.326 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc16ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:00.326 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:00.326 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc2cb16640 5 monclient: authenticate success, global_id 4568 2024-03-25T19:50:00.326 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc2cb16640 10 monclient: _renew_subs 2024-03-25T19:50:00.326 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc2cb16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:50:00.326 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc2cb16640 1 -- 172.21.15.81:0/118510269 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc2813ad70 con 0x7fcc280b1730 2024-03-25T19:50:00.326 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc2cb16640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:00.326 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc2cb16640 10 monclient: _renew_subs 2024-03-25T19:50:00.326 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc2cb16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:50:00.326 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc2cb16640 1 -- 172.21.15.81:0/118510269 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc28145330 con 0x7fcc280b1730 2024-03-25T19:50:00.326 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.325+0000 7fcc2cb16640 1 librados: init done 2024-03-25T19:50:00.327 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.326+0000 7fcc2cb16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:00.327 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.326+0000 7fcc2cb16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:00.327 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.326+0000 7fcc2cb16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:50:00.327 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.326+0000 7fcc2cb16640 1 -- 172.21.15.81:0/118510269 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc281401f0 con 0x7fcc280b1730 2024-03-25T19:50:00.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.629+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc18001c70 con 0x7fcc280b1730 2024-03-25T19:50:00.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.629+0000 7fcc16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:00.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.629+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fcc18003a60 con 0x7fcc280b1730 2024-03-25T19:50:00.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.629+0000 7fcc16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:00.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.629+0000 7fcc16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:00.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.629+0000 7fcc16ffd640 10 monclient: dump: 2024-03-25T19:50:00.629 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:00.629 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:00.629 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:00.629 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:00.629 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:00.629 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:00.629 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:00.629 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:00.630 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:00.630 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:00.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.629+0000 7fcc24d99640 4 set_mon_vals no callback set 2024-03-25T19:50:00.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.629+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fcc18003c40 con 0x7fcc280b1730 2024-03-25T19:50:00.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.630+0000 7fcc25d9b640 10 monclient: get_auth_request con 0x7fcc0806df70 auth_method 0 2024-03-25T19:50:00.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.630+0000 7fcc16ffd640 10 client.4568.objecter ms_handle_connect 0x7fcc0806df70 2024-03-25T19:50:00.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.636+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fcc180548b0 con 0x7fcc280b1730 2024-03-25T19:50:00.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.636+0000 7fcc16ffd640 10 client.4568.objecter ms_dispatch 0x7fcc28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:00.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.636+0000 7fcc16ffd640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:00.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.636+0000 7fcc16ffd640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:00.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.636+0000 7fcc16ffd640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-25T19:50:00.637 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.636+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc180503c0 con 0x7fcc280b1730 2024-03-25T19:50:00.637 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.637+0000 7fcc16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:00.637 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.637+0000 7fcc16ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:00.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.873+0000 7fcc2cb16640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:00.873 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.873+0000 7fcc2cb16640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:00.873 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.873+0000 7fcc2cb16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:50:00.873 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.873+0000 7fcc2cb16640 1 -- 172.21.15.81:0/118510269 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc280afbd0 con 0x7fcc280b1730 2024-03-25T19:50:00.913 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.913+0000 7fcc16ffd640 1 -- 172.21.15.81:0/118510269 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fcc180522f0 con 0x7fcc280b1730 2024-03-25T19:50:00.913 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.913+0000 7fcc16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:00.913 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.913+0000 7fcc16ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:00.913 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:50:00.914 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:00.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.916+0000 7fcc2cb16640 10 librados: watch_flush enter 2024-03-25T19:50:00.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.916+0000 7fcc2cb16640 10 librados: watch_flush exit 2024-03-25T19:50:00.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.916+0000 7fcc2cb16640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-25T19:50:00.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.916+0000 7fcc2cb16640 10 client.4568.objecter successfully canceled tick 2024-03-25T19:50:00.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.916+0000 7fcc2cb16640 1 -- 172.21.15.81:0/118510269 >> v1:172.21.15.98:6800/3980829775 conn(0x7fcc0806df70 legacy=0x7fcc08070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:00.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.916+0000 7fcc2cb16640 10 monclient: shutdown 2024-03-25T19:50:00.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.916+0000 7fcc2cb16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:00.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.916+0000 7fcc2cb16640 1 -- 172.21.15.81:0/118510269 >> v1:172.21.15.81:6790/0 conn(0x7fcc280b1730 legacy=0x7fcc2813b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:00.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.916+0000 7fcc2cb16640 1 -- 172.21.15.81:0/118510269 shutdown_connections 2024-03-25T19:50:00.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.916+0000 7fcc2cb16640 1 -- 172.21.15.81:0/118510269 >> 172.21.15.81:0/118510269 conn(0x7fcc280617a0 msgr2=0x7fcc2813c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:00.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.916+0000 7fcc2cb16640 1 -- 172.21.15.81:0/118510269 shutdown_connections 2024-03-25T19:50:00.917 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.916+0000 7fcc2cb16640 1 -- 172.21.15.81:0/118510269 wait complete. 2024-03-25T19:50:00.917 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:00.916+0000 7fcc2cb16640 1 librados: shutdown 2024-03-25T19:50:00.927 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:03.170 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:08:25.811 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:03.227 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:08:25.869 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:03.251 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:08:25.893 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:03.278 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:08:25.912 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:03.285 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:08:25.920 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:03.298 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:08:25.940 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:03.303 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:08:25.945 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:03.313 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:08:25.947 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:03.331 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:08:25.973 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:07.734 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:07.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.851+0000 7f3ba6b32640 10 monclient: get_monmap_and_config 2024-03-25T19:50:07.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.851+0000 7f3ba6b32640 10 monclient: build_initial_monmap 2024-03-25T19:50:07.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.851+0000 7f3ba6b32640 10 monclient: monmap: 2024-03-25T19:50:07.851 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:50:07.851 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:07.852 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:50:07.852581+0000 2024-03-25T19:50:07.852 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:50:07.852581+0000 2024-03-25T19:50:07.852 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:07.852 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:07.852 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:50:07.852 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:50:07.852 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:50:07.852 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:07.853 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.852+0000 7f3ba6b32640 1 Processor -- start 2024-03-25T19:50:07.853 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.852+0000 7f3ba6b32640 1 -- start start 2024-03-25T19:50:07.853 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.852+0000 7f3ba6b32640 10 monclient: init 2024-03-25T19:50:07.854 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.853+0000 7f3ba6b32640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:07.854 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.853+0000 7f3ba6b32640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:07.854 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.853+0000 7f3ba6b32640 10 monclient(hunting): picked mon.noname-a con 0x7f3ba00a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:50:07.854 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.853+0000 7f3ba6b32640 10 monclient(hunting): picked mon.noname-b con 0x7f3ba013f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:50:07.854 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.853+0000 7f3ba6b32640 10 monclient(hunting): picked mon.noname-c con 0x7f3ba01430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:50:07.854 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.853+0000 7f3ba6b32640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba0142520 con 0x7f3ba00a98f0 2024-03-25T19:50:07.854 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.853+0000 7f3ba6b32640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba00b28a0 con 0x7f3ba01430c0 2024-03-25T19:50:07.854 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.853+0000 7f3ba6b32640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba00b2a30 con 0x7f3ba013f710 2024-03-25T19:50:07.854 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.853+0000 7f3ba6b32640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:07.854 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.853+0000 7f3ba6b32640 10 monclient(hunting): authenticate will time out at 4442.935727s 2024-03-25T19:50:07.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.862+0000 7f3ba48ce640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f3ba00a98f0 0x7f3ba00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:59274/0 (socket says 172.21.15.81:59274) 2024-03-25T19:50:07.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.862+0000 7f3ba48ce640 1 -- 172.21.15.81:0/825983601 learned_addr learned my addr 172.21.15.81:0/825983601 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:50:07.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.877+0000 7f3b9f7fe640 1 -- 172.21.15.81:0/825983601 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467741014 0 0) 0x7f3ba0142520 con 0x7f3ba00a98f0 2024-03-25T19:50:07.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.877+0000 7f3b9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:07.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.877+0000 7f3b9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:07.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.877+0000 7f3b9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:07.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.877+0000 7f3b9f7fe640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:50:07.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.877+0000 7f3b9f7fe640 1 -- 172.21.15.81:0/825983601 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b90003520 con 0x7f3ba00a98f0 2024-03-25T19:50:07.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.881+0000 7f3b9f7fe640 1 -- 172.21.15.81:0/825983601 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994372656 0 0) 0x7f3ba00b28a0 con 0x7f3ba01430c0 2024-03-25T19:50:07.881 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.881+0000 7f3b9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:07.881 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.881+0000 7f3b9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:07.881 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.881+0000 7f3b9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:07.881 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.881+0000 7f3b9f7fe640 10 monclient(hunting): my global_id is 4588 2024-03-25T19:50:07.881 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.881+0000 7f3b9f7fe640 1 -- 172.21.15.81:0/825983601 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ba0142520 con 0x7f3ba01430c0 2024-03-25T19:50:07.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.883+0000 7f3b9f7fe640 1 -- 172.21.15.81:0/825983601 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396010836 0 0) 0x7f3ba00b2a30 con 0x7f3ba013f710 2024-03-25T19:50:07.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.883+0000 7f3b9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:07.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.883+0000 7f3b9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:07.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.883+0000 7f3b9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:07.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.883+0000 7f3b9f7fe640 10 monclient(hunting): my global_id is 4571 2024-03-25T19:50:07.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.883+0000 7f3b9f7fe640 1 -- 172.21.15.81:0/825983601 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ba00b28a0 con 0x7f3ba013f710 2024-03-25T19:50:07.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.887+0000 7f3b9f7fe640 1 -- 172.21.15.81:0/825983601 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2549625491 0 0) 0x7f3ba0142520 con 0x7f3ba01430c0 2024-03-25T19:50:07.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.887+0000 7f3b9f7fe640 1 -- 172.21.15.81:0/825983601 >> v1:172.21.15.81:6790/0 conn(0x7f3ba013f710 legacy=0x7f3ba01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:07.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.887+0000 7f3b9f7fe640 1 -- 172.21.15.81:0/825983601 >> v1:172.21.15.81:6789/0 conn(0x7f3ba00a98f0 legacy=0x7f3ba00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:07.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.887+0000 7f3b9f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:50:07.887 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.887+0000 7f3b9f7fe640 1 monclient: found mon.noname-c 2024-03-25T19:50:07.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.887+0000 7f3b9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:07.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.887+0000 7f3b9f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:50:07.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.887+0000 7f3b9f7fe640 1 -- 172.21.15.81:0/825983601 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba00b44c0 con 0x7f3ba01430c0 2024-03-25T19:50:07.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.887+0000 7f3b9f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:50:07.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.887+0000 7f3b9f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:50:07.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.887+0000 7f3b9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:07.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.888+0000 7f3b9f7fe640 1 -- 172.21.15.81:0/825983601 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3b980019f0 con 0x7f3ba01430c0 2024-03-25T19:50:07.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.888+0000 7f3ba6b32640 5 monclient: authenticate success, global_id 4588 2024-03-25T19:50:07.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.888+0000 7f3ba6b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:07.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.888+0000 7f3b9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:07.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.888+0000 7f3b9f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:07.888 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.888+0000 7f3b9f7fe640 10 monclient: dump: 2024-03-25T19:50:07.888 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:07.888 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:07.888 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:07.888 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:07.889 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:07.889 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:07.889 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:07.889 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:07.889 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:07.889 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:07.889 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.888+0000 7f3ba6b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:07.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.900+0000 7f3b9f7fe640 1 -- 172.21.15.81:0/825983601 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b98001b50 con 0x7f3ba01430c0 2024-03-25T19:50:07.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.900+0000 7f3b9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:07.900 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.900+0000 7f3b9f7fe640 1 -- 172.21.15.81:0/825983601 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3b98001f60 con 0x7f3ba01430c0 2024-03-25T19:50:07.900 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.900+0000 7f3ba6b32640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:07.900 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.900+0000 7f3ba6b32640 4 set_mon_vals no callback set 2024-03-25T19:50:07.900 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.900+0000 7f3b9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:07.900 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.900+0000 7f3b9f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:07.900 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.900+0000 7f3b9f7fe640 10 monclient: dump: 2024-03-25T19:50:07.900 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:07.900 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:07.900 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:07.900 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:07.900 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:07.900 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:07.900 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:07.901 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:07.901 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:07.901 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:07.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.901+0000 7f3ba6b32640 10 monclient: shutdown 2024-03-25T19:50:07.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.901+0000 7f3ba6b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:07.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.901+0000 7f3ba6b32640 1 -- 172.21.15.81:0/825983601 >> v1:172.21.15.98:6789/0 conn(0x7f3ba01430c0 legacy=0x7f3ba014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:07.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.901+0000 7f3ba6b32640 1 -- 172.21.15.81:0/825983601 shutdown_connections 2024-03-25T19:50:07.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.901+0000 7f3ba6b32640 1 -- 172.21.15.81:0/825983601 >> 172.21.15.81:0/825983601 conn(0x7f3ba00b7ea0 msgr2=0x7f3ba00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:07.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.901+0000 7f3ba6b32640 1 -- 172.21.15.81:0/825983601 shutdown_connections 2024-03-25T19:50:07.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.901+0000 7f3ba6b32640 1 -- 172.21.15.81:0/825983601 wait complete. 2024-03-25T19:50:07.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.902+0000 7f3ba6b32640 10 monclient: build_initial_monmap 2024-03-25T19:50:07.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.902+0000 7f3ba6b32640 10 monclient: monmap: 2024-03-25T19:50:07.903 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:50:07.903 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:07.903 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:50:07.903 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:50:07.904 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:07.904 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:50:07.904 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:50:07.904 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:50:07.904 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:50:07.904 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:07.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.902+0000 7f3ba6b32640 1 librados: starting msgr at 2024-03-25T19:50:07.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.902+0000 7f3ba6b32640 1 librados: starting objecter 2024-03-25T19:50:07.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.902+0000 7f3ba6b32640 1 Processor -- start 2024-03-25T19:50:07.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.902+0000 7f3ba6b32640 1 -- start start 2024-03-25T19:50:07.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.902+0000 7f3ba6b32640 1 librados: setting wanted keys 2024-03-25T19:50:07.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.902+0000 7f3ba6b32640 1 librados: calling monclient init 2024-03-25T19:50:07.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.902+0000 7f3ba6b32640 10 monclient: init 2024-03-25T19:50:07.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.903+0000 7f3ba6b32640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:07.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.903+0000 7f3ba6b32640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:07.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.903+0000 7f3ba6b32640 10 monclient(hunting): picked mon.noname-a con 0x7f3ba00a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:50:07.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.903+0000 7f3ba6b32640 10 monclient(hunting): picked mon.noname-c con 0x7f3ba013f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:50:07.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.903+0000 7f3ba6b32640 10 monclient(hunting): picked mon.noname-b con 0x7f3ba01430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:50:07.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.903+0000 7f3ba6b32640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba00ad460 con 0x7f3ba00a98f0 2024-03-25T19:50:07.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.903+0000 7f3ba6b32640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba00b1670 con 0x7f3ba01430c0 2024-03-25T19:50:07.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.903+0000 7f3ba6b32640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba01d7a90 con 0x7f3ba013f710 2024-03-25T19:50:07.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.903+0000 7f3ba6b32640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:07.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.903+0000 7f3ba6b32640 10 monclient(hunting): authenticate will time out at 4442.985316s 2024-03-25T19:50:07.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.911+0000 7f3b9ffff640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f3ba013f710 0x7f3ba01d7380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:43106/0 (socket says 172.21.15.81:43106) 2024-03-25T19:50:07.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.911+0000 7f3b9ffff640 1 -- 172.21.15.81:0/3037433107 learned_addr learned my addr 172.21.15.81:0/3037433107 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:50:07.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.912+0000 7f3b9d7fa640 10 client.?.objecter ms_handle_connect 0x7f3ba013f710 2024-03-25T19:50:07.913 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.912+0000 7f3b9d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:07.918 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.918+0000 7f3b9d7fa640 10 client.?.objecter ms_handle_connect 0x7f3ba01430c0 2024-03-25T19:50:07.918 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.918+0000 7f3b9d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:07.918 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.918+0000 7f3b9d7fa640 10 client.?.objecter ms_handle_connect 0x7f3ba00a98f0 2024-03-25T19:50:07.918 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.918+0000 7f3b9d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:07.924 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.925+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655078066 0 0) 0x7f3b90003520 con 0x7f3ba00a98f0 2024-03-25T19:50:07.924 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.925+0000 7f3b9d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:07.924 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.925+0000 7f3b9d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:07.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.925+0000 7f3b9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:07.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.925+0000 7f3b9d7fa640 10 monclient(hunting): my global_id is 4602 2024-03-25T19:50:07.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.925+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b84004eb0 con 0x7f3ba00a98f0 2024-03-25T19:50:07.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.927+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995443969 0 0) 0x7f3ba00b1670 con 0x7f3ba01430c0 2024-03-25T19:50:07.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.927+0000 7f3b9d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:07.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.927+0000 7f3b9d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:07.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.927+0000 7f3b9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:07.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.927+0000 7f3b9d7fa640 10 monclient(hunting): my global_id is 4591 2024-03-25T19:50:07.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.927+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b90003520 con 0x7f3ba01430c0 2024-03-25T19:50:07.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.932+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107031714 0 0) 0x7f3ba01d7a90 con 0x7f3ba013f710 2024-03-25T19:50:07.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.932+0000 7f3b9d7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:07.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.932+0000 7f3b9d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:07.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.932+0000 7f3b9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:07.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.932+0000 7f3b9d7fa640 10 monclient(hunting): my global_id is 4574 2024-03-25T19:50:07.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.932+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ba00b1670 con 0x7f3ba013f710 2024-03-25T19:50:07.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.933+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1623606923 0 0) 0x7f3b84004eb0 con 0x7f3ba00a98f0 2024-03-25T19:50:07.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.933+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ba01d7a90 con 0x7f3ba00a98f0 2024-03-25T19:50:07.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.933+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3b940017b0 con 0x7f3ba00a98f0 2024-03-25T19:50:07.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.933+0000 7f3b9d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:07.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.933+0000 7f3b9d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:07.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.933+0000 7f3b9d7fa640 10 monclient(hunting): dump: 2024-03-25T19:50:07.934 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:07.934 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:07.934 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:07.934 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:07.934 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:07.934 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:07.934 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:07.934 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:07.934 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:07.934 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:07.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.933+0000 7f3b9d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:07.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.934+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3839138694 0 0) 0x7f3b90003520 con 0x7f3ba01430c0 2024-03-25T19:50:07.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.934+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b84004eb0 con 0x7f3ba01430c0 2024-03-25T19:50:07.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.934+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3b98001e60 con 0x7f3ba01430c0 2024-03-25T19:50:07.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.934+0000 7f3b9d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:07.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.934+0000 7f3b9d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:07.935 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.934+0000 7f3b9d7fa640 10 monclient(hunting): dump: 2024-03-25T19:50:07.935 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:07.935 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:07.935 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:07.935 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:07.935 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:07.935 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:07.935 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:07.935 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:07.935 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:07.936 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:07.938 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3075902399 0 0) 0x7f3ba01d7a90 con 0x7f3ba00a98f0 2024-03-25T19:50:07.938 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 >> v1:172.21.15.81:6790/0 conn(0x7f3ba013f710 legacy=0x7f3ba01d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:07.938 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 >> v1:172.21.15.98:6789/0 conn(0x7f3ba01430c0 legacy=0x7f3ba014d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:07.938 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3b9d7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:50:07.938 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3b9d7fa640 1 monclient: found mon.a 2024-03-25T19:50:07.938 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3b9d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:07.938 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3b9d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:50:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba014ece0 con 0x7f3ba00a98f0 2024-03-25T19:50:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3b9d7fa640 10 monclient: _finish_auth 0 2024-03-25T19:50:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3b9d7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:50:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3b9d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3ba6b32640 5 monclient: authenticate success, global_id 4602 2024-03-25T19:50:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3ba6b32640 10 monclient: _renew_subs 2024-03-25T19:50:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3ba6b32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:50:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3ba6b32640 1 -- 172.21.15.81:0/3037433107 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba014dea0 con 0x7f3ba00a98f0 2024-03-25T19:50:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3ba6b32640 10 client.4602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3ba6b32640 10 monclient: _renew_subs 2024-03-25T19:50:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3ba6b32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:50:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3ba6b32640 1 -- 172.21.15.81:0/3037433107 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba00b13e0 con 0x7f3ba00a98f0 2024-03-25T19:50:07.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.938+0000 7f3ba6b32640 1 librados: init done 2024-03-25T19:50:07.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.939+0000 7f3ba6b32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:07.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.939+0000 7f3ba6b32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:07.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.939+0000 7f3ba6b32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:50:07.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.939+0000 7f3ba6b32640 1 -- 172.21.15.81:0/3037433107 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba00627b0 con 0x7f3ba00a98f0 2024-03-25T19:50:07.941 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.941+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b94001bc0 con 0x7f3ba00a98f0 2024-03-25T19:50:07.941 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.941+0000 7f3b9d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:07.941 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:07.941+0000 7f3b9effd640 4 set_mon_vals no callback set 2024-03-25T19:50:08.242 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.243+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f3b94004980 con 0x7f3ba00a98f0 2024-03-25T19:50:08.243 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.243+0000 7f3b9d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:08.243 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.243+0000 7f3b9d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:08.243 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.243+0000 7f3b9d7fa640 10 monclient: dump: 2024-03-25T19:50:08.243 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:08.243 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:08.243 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:08.243 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:08.243 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:08.243 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:08.243 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:08.243 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:08.243 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:08.243 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:08.244 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.244+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f3b9401a950 con 0x7f3ba00a98f0 2024-03-25T19:50:08.244 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.244+0000 7f3b9ffff640 10 monclient: get_auth_request con 0x7f3b8406df10 auth_method 0 2024-03-25T19:50:08.244 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.244+0000 7f3b9d7fa640 10 client.4602.objecter ms_handle_connect 0x7f3b8406df10 2024-03-25T19:50:08.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.253+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f3b94054560 con 0x7f3ba00a98f0 2024-03-25T19:50:08.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.253+0000 7f3b9d7fa640 10 client.4602.objecter ms_dispatch 0x7f3ba0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:08.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.253+0000 7f3b9d7fa640 3 client.4602.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:08.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.253+0000 7f3b9d7fa640 3 client.4602.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:08.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.253+0000 7f3b9d7fa640 20 client.4602.objecter dump_active .. 0 homeless 2024-03-25T19:50:08.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.253+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3b9404fea0 con 0x7f3ba00a98f0 2024-03-25T19:50:08.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.253+0000 7f3b9d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:08.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.253+0000 7f3b9d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:08.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.500+0000 7f3ba6b32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:08.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.500+0000 7f3ba6b32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:08.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.500+0000 7f3ba6b32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:50:08.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.500+0000 7f3ba6b32640 1 -- 172.21.15.81:0/3037433107 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ba013b430 con 0x7f3ba00a98f0 2024-03-25T19:50:08.584 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.584+0000 7f3b9d7fa640 1 -- 172.21.15.81:0/3037433107 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f3b94051fa0 con 0x7f3ba00a98f0 2024-03-25T19:50:08.584 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.584+0000 7f3b9d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:08.584 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.584+0000 7f3b9d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:08.584 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:50:08.584 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:08.586 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.586+0000 7f3ba6b32640 10 librados: watch_flush enter 2024-03-25T19:50:08.587 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.586+0000 7f3ba6b32640 10 librados: watch_flush exit 2024-03-25T19:50:08.587 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.586+0000 7f3ba6b32640 20 client.4602.objecter shutdown clearing up homeless session... 2024-03-25T19:50:08.587 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.586+0000 7f3ba6b32640 10 client.4602.objecter successfully canceled tick 2024-03-25T19:50:08.587 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.586+0000 7f3ba6b32640 1 -- 172.21.15.81:0/3037433107 >> v1:172.21.15.98:6800/3980829775 conn(0x7f3b8406df10 legacy=0x7f3b840703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:08.587 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.586+0000 7f3ba6b32640 10 monclient: shutdown 2024-03-25T19:50:08.587 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.586+0000 7f3ba6b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:08.587 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.586+0000 7f3ba6b32640 1 -- 172.21.15.81:0/3037433107 >> v1:172.21.15.81:6789/0 conn(0x7f3ba00a98f0 legacy=0x7f3ba01d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:08.587 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.587+0000 7f3ba6b32640 1 -- 172.21.15.81:0/3037433107 shutdown_connections 2024-03-25T19:50:08.587 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.587+0000 7f3ba6b32640 1 -- 172.21.15.81:0/3037433107 >> 172.21.15.81:0/3037433107 conn(0x7f3ba00b7ea0 msgr2=0x7f3ba0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:08.587 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.587+0000 7f3ba6b32640 1 -- 172.21.15.81:0/3037433107 shutdown_connections 2024-03-25T19:50:08.587 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.587+0000 7f3ba6b32640 1 -- 172.21.15.81:0/3037433107 wait complete. 2024-03-25T19:50:08.587 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:08.587+0000 7f3ba6b32640 1 librados: shutdown 2024-03-25T19:50:08.597 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:15.403 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:15.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.512+0000 7f93feff6640 10 monclient: get_monmap_and_config 2024-03-25T19:50:15.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.512+0000 7f93feff6640 10 monclient: build_initial_monmap 2024-03-25T19:50:15.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.512+0000 7f93feff6640 10 monclient: monmap: 2024-03-25T19:50:15.511 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:50:15.512 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:15.512 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:50:15.512998+0000 2024-03-25T19:50:15.512 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:50:15.512998+0000 2024-03-25T19:50:15.512 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:15.512 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:15.512 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:50:15.512 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:50:15.512 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:50:15.512 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:15.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.512+0000 7f93feff6640 1 Processor -- start 2024-03-25T19:50:15.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.512+0000 7f93feff6640 1 -- start start 2024-03-25T19:50:15.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.512+0000 7f93feff6640 10 monclient: init 2024-03-25T19:50:15.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.513+0000 7f93feff6640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:15.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.513+0000 7f93feff6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:15.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.513+0000 7f93feff6640 10 monclient(hunting): picked mon.noname-c con 0x7f93f80b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:50:15.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.513+0000 7f93feff6640 10 monclient(hunting): picked mon.noname-b con 0x7f93f80b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:50:15.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.513+0000 7f93feff6640 10 monclient(hunting): picked mon.noname-a con 0x7f93f814be90 addr v1:172.21.15.81:6789/0 2024-03-25T19:50:15.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.513+0000 7f93feff6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f80af940 con 0x7f93f814be90 2024-03-25T19:50:15.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.513+0000 7f93feff6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f80b0a70 con 0x7f93f80b4d60 2024-03-25T19:50:15.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.513+0000 7f93feff6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f80b0d50 con 0x7f93f80b1730 2024-03-25T19:50:15.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.513+0000 7f93feff6640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:15.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.513+0000 7f93feff6640 10 monclient(hunting): authenticate will time out at 4450.595268s 2024-03-25T19:50:15.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.513+0000 7f93f7fff640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f93f80b1730 0x7f93f80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:47980/0 (socket says 172.21.15.81:47980) 2024-03-25T19:50:15.514 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.513+0000 7f93f7fff640 1 -- 172.21.15.81:0/58625962 learned_addr learned my addr 172.21.15.81:0/58625962 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:50:15.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.555+0000 7f93f77fe640 1 -- 172.21.15.81:0/58625962 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214458643 0 0) 0x7f93f80af940 con 0x7f93f814be90 2024-03-25T19:50:15.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.555+0000 7f93f77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:15.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.555+0000 7f93f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:15.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.555+0000 7f93f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:15.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.555+0000 7f93f77fe640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:50:15.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.555+0000 7f93f77fe640 1 -- 172.21.15.81:0/58625962 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93e8003520 con 0x7f93f814be90 2024-03-25T19:50:15.573 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.573+0000 7f93f77fe640 1 -- 172.21.15.81:0/58625962 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1207096014 0 0) 0x7f93e8003520 con 0x7f93f814be90 2024-03-25T19:50:15.573 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.573+0000 7f93f77fe640 1 -- 172.21.15.81:0/58625962 >> v1:172.21.15.81:6790/0 conn(0x7f93f80b1730 legacy=0x7f93f80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:15.573 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.573+0000 7f93f77fe640 1 -- 172.21.15.81:0/58625962 >> v1:172.21.15.98:6789/0 conn(0x7f93f80b4d60 legacy=0x7f93f813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:15.573 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.573+0000 7f93f77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:50:15.573 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.573+0000 7f93f77fe640 1 monclient: found mon.noname-a 2024-03-25T19:50:15.573 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.573+0000 7f93f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:15.573 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.573+0000 7f93f77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:50:15.573 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.573+0000 7f93f77fe640 1 -- 172.21.15.81:0/58625962 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f80b1030 con 0x7f93f814be90 2024-03-25T19:50:15.573 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.573+0000 7f93f77fe640 10 monclient: _finish_auth 0 2024-03-25T19:50:15.573 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.573+0000 7f93f77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:50:15.573 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.573+0000 7f93f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:15.574 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.573+0000 7f93f77fe640 1 -- 172.21.15.81:0/58625962 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f93f0001a10 con 0x7f93f814be90 2024-03-25T19:50:15.574 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.573+0000 7f93f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:15.574 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.573+0000 7f93f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:15.574 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.573+0000 7f93f77fe640 10 monclient: dump: 2024-03-25T19:50:15.574 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:15.574 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:15.574 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:15.574 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:15.574 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:15.574 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:15.574 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:15.574 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:15.575 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:15.575 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:15.575 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.573+0000 7f93feff6640 5 monclient: authenticate success, global_id 4605 2024-03-25T19:50:15.575 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.573+0000 7f93feff6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:15.591 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.592+0000 7f93f77fe640 1 -- 172.21.15.81:0/58625962 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93f0001b70 con 0x7f93f814be90 2024-03-25T19:50:15.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.592+0000 7f93f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:15.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.592+0000 7f93f77fe640 1 -- 172.21.15.81:0/58625962 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f93f0001fc0 con 0x7f93f814be90 2024-03-25T19:50:15.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.592+0000 7f93f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:15.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.592+0000 7f93f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:15.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.592+0000 7f93f77fe640 10 monclient: dump: 2024-03-25T19:50:15.592 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:15.592 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:15.592 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:15.592 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:15.592 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:15.592 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:15.593 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:15.593 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:15.593 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:15.593 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:15.593 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.592+0000 7f93feff6640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:15.593 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.592+0000 7f93feff6640 4 set_mon_vals no callback set 2024-03-25T19:50:15.595 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.593+0000 7f93feff6640 10 monclient: shutdown 2024-03-25T19:50:15.595 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.593+0000 7f93feff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:15.595 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.593+0000 7f93feff6640 1 -- 172.21.15.81:0/58625962 >> v1:172.21.15.81:6789/0 conn(0x7f93f814be90 legacy=0x7f93f814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:15.595 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.593+0000 7f93feff6640 1 -- 172.21.15.81:0/58625962 shutdown_connections 2024-03-25T19:50:15.595 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.593+0000 7f93feff6640 1 -- 172.21.15.81:0/58625962 >> 172.21.15.81:0/58625962 conn(0x7f93f80617a0 msgr2=0x7f93f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:15.595 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.593+0000 7f93feff6640 1 -- 172.21.15.81:0/58625962 shutdown_connections 2024-03-25T19:50:15.595 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.593+0000 7f93feff6640 1 -- 172.21.15.81:0/58625962 wait complete. 2024-03-25T19:50:15.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.594+0000 7f93feff6640 10 monclient: build_initial_monmap 2024-03-25T19:50:15.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.594+0000 7f93feff6640 10 monclient: monmap: 2024-03-25T19:50:15.596 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:50:15.596 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:15.596 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:50:15.596 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:50:15.596 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:15.596 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:50:15.596 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:50:15.596 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:50:15.596 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:50:15.596 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:15.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.594+0000 7f93feff6640 1 librados: starting msgr at 2024-03-25T19:50:15.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.594+0000 7f93feff6640 1 librados: starting objecter 2024-03-25T19:50:15.596 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.594+0000 7f93feff6640 1 Processor -- start 2024-03-25T19:50:15.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.594+0000 7f93feff6640 1 -- start start 2024-03-25T19:50:15.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.594+0000 7f93feff6640 1 librados: setting wanted keys 2024-03-25T19:50:15.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.594+0000 7f93feff6640 1 librados: calling monclient init 2024-03-25T19:50:15.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.594+0000 7f93feff6640 10 monclient: init 2024-03-25T19:50:15.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.595+0000 7f93feff6640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:15.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.595+0000 7f93feff6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:50:15.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.595+0000 7f93feff6640 10 monclient(hunting): picked mon.noname-b con 0x7f93f80b1730 addr v1:172.21.15.98:6789/0 2024-03-25T19:50:15.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.595+0000 7f93feff6640 10 monclient(hunting): picked mon.noname-a con 0x7f93f80b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:50:15.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.595+0000 7f93feff6640 10 monclient(hunting): picked mon.noname-c con 0x7f93f814be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:50:15.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.595+0000 7f93feff6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f8140380 con 0x7f93f80b4d60 2024-03-25T19:50:15.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.595+0000 7f93feff6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f8140660 con 0x7f93f80b1730 2024-03-25T19:50:15.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.595+0000 7f93feff6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f8141e30 con 0x7f93f814be90 2024-03-25T19:50:15.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.595+0000 7f93feff6640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:15.597 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.595+0000 7f93feff6640 10 monclient(hunting): authenticate will time out at 4450.677418s 2024-03-25T19:50:15.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.604+0000 7f93f7fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f93f80b4d60 0x7f93f8142530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:36032/0 (socket says 172.21.15.81:36032) 2024-03-25T19:50:15.604 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.605+0000 7f93f7fff640 1 -- 172.21.15.81:0/847811805 learned_addr learned my addr 172.21.15.81:0/847811805 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:50:15.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.605+0000 7f93f57fa640 10 client.?.objecter ms_handle_connect 0x7f93f80b4d60 2024-03-25T19:50:15.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.605+0000 7f93f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:15.612 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.612+0000 7f93f57fa640 10 client.?.objecter ms_handle_connect 0x7f93f80b1730 2024-03-25T19:50:15.612 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.612+0000 7f93f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:15.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.621+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447809078 0 0) 0x7f93f8140380 con 0x7f93f80b4d60 2024-03-25T19:50:15.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.621+0000 7f93f57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:15.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.621+0000 7f93f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:15.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.621+0000 7f93f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:15.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.621+0000 7f93f57fa640 10 monclient(hunting): my global_id is 4608 2024-03-25T19:50:15.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.621+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93dc004eb0 con 0x7f93f80b4d60 2024-03-25T19:50:15.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.621+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926638963 0 0) 0x7f93f8140660 con 0x7f93f80b1730 2024-03-25T19:50:15.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.621+0000 7f93f57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:15.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.621+0000 7f93f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:15.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.621+0000 7f93f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:15.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.621+0000 7f93f57fa640 10 monclient(hunting): my global_id is 4594 2024-03-25T19:50:15.621 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.621+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93f8140380 con 0x7f93f80b1730 2024-03-25T19:50:15.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.628+0000 7f93f57fa640 10 client.?.objecter ms_handle_connect 0x7f93f814be90 2024-03-25T19:50:15.627 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.628+0000 7f93f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:15.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.638+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2055369861 0 0) 0x7f93f8140380 con 0x7f93f80b1730 2024-03-25T19:50:15.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.638+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93f8140660 con 0x7f93f80b1730 2024-03-25T19:50:15.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.638+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f93ec0018a0 con 0x7f93f80b1730 2024-03-25T19:50:15.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.638+0000 7f93f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:15.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.638+0000 7f93f57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:15.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.638+0000 7f93f57fa640 10 monclient(hunting): dump: 2024-03-25T19:50:15.638 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:15.638 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:15.638 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:15.638 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:15.638 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:15.639 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:15.639 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:15.639 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:15.639 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:15.639 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:15.639 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.638+0000 7f93f57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:15.639 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.639+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3540655079 0 0) 0x7f93dc004eb0 con 0x7f93f80b4d60 2024-03-25T19:50:15.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.639+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93f8140380 con 0x7f93f80b4d60 2024-03-25T19:50:15.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.639+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f93e40018b0 con 0x7f93f80b4d60 2024-03-25T19:50:15.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.639+0000 7f93f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:15.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.639+0000 7f93f57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:15.640 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.639+0000 7f93f57fa640 10 monclient(hunting): dump: 2024-03-25T19:50:15.640 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:15.640 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:15.640 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:15.640 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:15.640 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:15.640 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:15.640 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:15.640 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:15.640 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:15.640 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:15.642 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.642+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212829846 0 0) 0x7f93f8141e30 con 0x7f93f814be90 2024-03-25T19:50:15.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.642+0000 7f93f57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:15.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.642+0000 7f93f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:15.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.642+0000 7f93f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:15.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.642+0000 7f93f57fa640 10 monclient(hunting): my global_id is 4580 2024-03-25T19:50:15.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.642+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93dc004eb0 con 0x7f93f814be90 2024-03-25T19:50:15.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1345444820 0 0) 0x7f93f8140660 con 0x7f93f80b1730 2024-03-25T19:50:15.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 >> v1:172.21.15.81:6790/0 conn(0x7f93f814be90 legacy=0x7f93f81e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:15.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 >> v1:172.21.15.81:6789/0 conn(0x7f93f80b4d60 legacy=0x7f93f8142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:15.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93f57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:50:15.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93f57fa640 1 monclient: found mon.b 2024-03-25T19:50:15.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93f57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:15.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93f57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:15.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f8142110 con 0x7f93f80b1730 2024-03-25T19:50:15.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93f57fa640 10 monclient: _finish_auth 0 2024-03-25T19:50:15.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93f57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:50:15.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:15.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93feff6640 5 monclient: authenticate success, global_id 4594 2024-03-25T19:50:15.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93feff6640 10 monclient: _renew_subs 2024-03-25T19:50:15.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93feff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:15.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93feff6640 1 -- 172.21.15.81:0/847811805 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f8140e60 con 0x7f93f80b1730 2024-03-25T19:50:15.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93feff6640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:15.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93feff6640 10 monclient: _renew_subs 2024-03-25T19:50:15.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93feff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:15.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93feff6640 1 -- 172.21.15.81:0/847811805 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f8142c40 con 0x7f93f80b1730 2024-03-25T19:50:15.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.643+0000 7f93feff6640 1 librados: init done 2024-03-25T19:50:15.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.645+0000 7f93feff6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:15.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.645+0000 7f93feff6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:15.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.645+0000 7f93feff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:15.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.645+0000 7f93feff6640 1 -- 172.21.15.81:0/847811805 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f80627b0 con 0x7f93f80b1730 2024-03-25T19:50:15.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.925+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93ec0035f0 con 0x7f93f80b1730 2024-03-25T19:50:15.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.925+0000 7f93f57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:15.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.925+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f93ec0039c0 con 0x7f93f80b1730 2024-03-25T19:50:15.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.925+0000 7f93f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:15.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.925+0000 7f93f57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:15.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.925+0000 7f93f57fa640 10 monclient: dump: 2024-03-25T19:50:15.925 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:15.925 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:15.926 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:15.926 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:15.926 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:15.926 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:15.926 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:15.926 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:15.926 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:15.926 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:15.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.925+0000 7f93f6ffd640 4 set_mon_vals no callback set 2024-03-25T19:50:15.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.926+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f93ec003be0 con 0x7f93f80b1730 2024-03-25T19:50:15.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.927+0000 7f93f7fff640 10 monclient: get_auth_request con 0x7f93dc06df50 auth_method 0 2024-03-25T19:50:15.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.927+0000 7f93f57fa640 10 client.4594.objecter ms_handle_connect 0x7f93dc06df50 2024-03-25T19:50:15.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.933+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f93ec053570 con 0x7f93f80b1730 2024-03-25T19:50:15.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.933+0000 7f93f57fa640 10 client.4594.objecter ms_dispatch 0x7f93f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:15.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.933+0000 7f93f57fa640 3 client.4594.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:15.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.933+0000 7f93f57fa640 3 client.4594.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:15.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.933+0000 7f93f57fa640 20 client.4594.objecter dump_active .. 0 homeless 2024-03-25T19:50:15.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.933+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f93ec04f080 con 0x7f93f80b1730 2024-03-25T19:50:15.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.933+0000 7f93f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:15.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:15.933+0000 7f93f57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:16.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.180+0000 7f93feff6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:16.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.180+0000 7f93feff6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:16.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.180+0000 7f93feff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:16.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.180+0000 7f93feff6640 1 -- 172.21.15.81:0/847811805 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93f80b5cf0 con 0x7f93f80b1730 2024-03-25T19:50:16.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.200+0000 7f93f57fa640 1 -- 172.21.15.81:0/847811805 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f93ec050fb0 con 0x7f93f80b1730 2024-03-25T19:50:16.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.200+0000 7f93f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:16.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.200+0000 7f93f57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:16.200 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:50:16.200 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:16.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.203+0000 7f93feff6640 10 librados: watch_flush enter 2024-03-25T19:50:16.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.203+0000 7f93feff6640 10 librados: watch_flush exit 2024-03-25T19:50:16.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.203+0000 7f93feff6640 20 client.4594.objecter shutdown clearing up homeless session... 2024-03-25T19:50:16.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.203+0000 7f93feff6640 10 client.4594.objecter successfully canceled tick 2024-03-25T19:50:16.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.203+0000 7f93feff6640 1 -- 172.21.15.81:0/847811805 >> v1:172.21.15.98:6800/3980829775 conn(0x7f93dc06df50 legacy=0x7f93dc070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:16.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.203+0000 7f93feff6640 10 monclient: shutdown 2024-03-25T19:50:16.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.203+0000 7f93feff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:16.203 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.203+0000 7f93feff6640 1 -- 172.21.15.81:0/847811805 >> v1:172.21.15.98:6789/0 conn(0x7f93f80b1730 legacy=0x7f93f8138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:16.204 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.203+0000 7f93feff6640 1 -- 172.21.15.81:0/847811805 shutdown_connections 2024-03-25T19:50:16.204 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.203+0000 7f93feff6640 1 -- 172.21.15.81:0/847811805 >> 172.21.15.81:0/847811805 conn(0x7f93f80617a0 msgr2=0x7f93f80b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:16.204 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.203+0000 7f93feff6640 1 -- 172.21.15.81:0/847811805 shutdown_connections 2024-03-25T19:50:16.204 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.203+0000 7f93feff6640 1 -- 172.21.15.81:0/847811805 wait complete. 2024-03-25T19:50:16.204 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:16.203+0000 7f93feff6640 1 librados: shutdown 2024-03-25T19:50:16.215 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:23.021 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:23.143 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.142+0000 7fe27b1fb640 10 monclient: get_monmap_and_config 2024-03-25T19:50:23.143 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.142+0000 7fe27b1fb640 10 monclient: build_initial_monmap 2024-03-25T19:50:23.143 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.142+0000 7fe27b1fb640 10 monclient: monmap: 2024-03-25T19:50:23.143 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:50:23.143 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:23.143 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:50:23.144380+0000 2024-03-25T19:50:23.143 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:50:23.144380+0000 2024-03-25T19:50:23.143 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:23.143 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:23.143 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:50:23.143 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:50:23.144 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:50:23.144 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:23.145 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.143+0000 7fe27b1fb640 1 Processor -- start 2024-03-25T19:50:23.145 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.143+0000 7fe27b1fb640 1 -- start start 2024-03-25T19:50:23.145 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.143+0000 7fe27b1fb640 10 monclient: init 2024-03-25T19:50:23.145 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.144+0000 7fe27b1fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:23.145 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.144+0000 7fe27b1fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:23.145 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.144+0000 7fe27b1fb640 10 monclient(hunting): picked mon.noname-a con 0x7fe2740b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:50:23.145 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.144+0000 7fe27b1fb640 10 monclient(hunting): picked mon.noname-b con 0x7fe2740b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:50:23.145 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.144+0000 7fe27b1fb640 10 monclient(hunting): picked mon.noname-c con 0x7fe27414be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:50:23.146 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.144+0000 7fe27b1fb640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2740af940 con 0x7fe2740b4d60 2024-03-25T19:50:23.146 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.144+0000 7fe27b1fb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2740b0a70 con 0x7fe27414be90 2024-03-25T19:50:23.146 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.144+0000 7fe27b1fb640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2740b0d50 con 0x7fe2740b1730 2024-03-25T19:50:23.146 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.144+0000 7fe27b1fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:23.146 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.144+0000 7fe27b1fb640 10 monclient(hunting): authenticate will time out at 4458.227336s 2024-03-25T19:50:23.151 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.151+0000 7fe278f97640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe2740b4d60 0x7fe27413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:44932/0 (socket says 172.21.15.81:44932) 2024-03-25T19:50:23.151 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.151+0000 7fe278f97640 1 -- 172.21.15.81:0/360879712 learned_addr learned my addr 172.21.15.81:0/360879712 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:50:23.160 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.160+0000 7fe2737fe640 1 -- 172.21.15.81:0/360879712 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234668269 0 0) 0x7fe2740af940 con 0x7fe2740b4d60 2024-03-25T19:50:23.160 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.160+0000 7fe2737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:23.161 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.160+0000 7fe2737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:23.161 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.160+0000 7fe2737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:23.161 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.160+0000 7fe2737fe640 10 monclient(hunting): my global_id is 4611 2024-03-25T19:50:23.161 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.160+0000 7fe2737fe640 1 -- 172.21.15.81:0/360879712 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe264003520 con 0x7fe2740b4d60 2024-03-25T19:50:23.173 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.173+0000 7fe2737fe640 1 -- 172.21.15.81:0/360879712 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965431752 0 0) 0x7fe2740b0a70 con 0x7fe27414be90 2024-03-25T19:50:23.173 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.173+0000 7fe2737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:23.173 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.173+0000 7fe2737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:23.173 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.173+0000 7fe2737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:23.173 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.173+0000 7fe2737fe640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:50:23.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.173+0000 7fe2737fe640 1 -- 172.21.15.81:0/360879712 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2740af940 con 0x7fe27414be90 2024-03-25T19:50:23.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.177+0000 7fe2737fe640 1 -- 172.21.15.81:0/360879712 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586299310 0 0) 0x7fe2740b0d50 con 0x7fe2740b1730 2024-03-25T19:50:23.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.177+0000 7fe2737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:23.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.177+0000 7fe2737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:23.177 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.177+0000 7fe2737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:23.178 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.177+0000 7fe2737fe640 10 monclient(hunting): my global_id is 4583 2024-03-25T19:50:23.178 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.177+0000 7fe2737fe640 1 -- 172.21.15.81:0/360879712 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2740b0a70 con 0x7fe2740b1730 2024-03-25T19:50:23.179 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.178+0000 7fe2737fe640 1 -- 172.21.15.81:0/360879712 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3295288560 0 0) 0x7fe2740af940 con 0x7fe27414be90 2024-03-25T19:50:23.179 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.178+0000 7fe2737fe640 1 -- 172.21.15.81:0/360879712 >> v1:172.21.15.81:6790/0 conn(0x7fe2740b1730 legacy=0x7fe2740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:23.179 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.178+0000 7fe2737fe640 1 -- 172.21.15.81:0/360879712 >> v1:172.21.15.81:6789/0 conn(0x7fe2740b4d60 legacy=0x7fe27413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:23.179 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.178+0000 7fe2737fe640 10 monclient: _finish_hunting 0 2024-03-25T19:50:23.179 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.178+0000 7fe2737fe640 1 monclient: found mon.noname-c 2024-03-25T19:50:23.179 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.178+0000 7fe2737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:23.179 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.178+0000 7fe2737fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:50:23.179 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.178+0000 7fe2737fe640 1 -- 172.21.15.81:0/360879712 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2740b1030 con 0x7fe27414be90 2024-03-25T19:50:23.179 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.178+0000 7fe2737fe640 10 monclient: _finish_auth 0 2024-03-25T19:50:23.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.178+0000 7fe2737fe640 10 monclient: _check_auth_tickets 2024-03-25T19:50:23.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.178+0000 7fe2737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:23.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.178+0000 7fe2737fe640 1 -- 172.21.15.81:0/360879712 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe26c0019f0 con 0x7fe27414be90 2024-03-25T19:50:23.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.178+0000 7fe2737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:23.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.178+0000 7fe2737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:23.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.178+0000 7fe2737fe640 10 monclient: dump: 2024-03-25T19:50:23.180 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:23.180 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:23.180 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:23.180 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:23.180 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:23.180 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:23.180 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:23.180 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:23.181 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:23.181 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:23.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.178+0000 7fe27b1fb640 5 monclient: authenticate success, global_id 4597 2024-03-25T19:50:23.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.178+0000 7fe27b1fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:23.190 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.189+0000 7fe2737fe640 1 -- 172.21.15.81:0/360879712 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe26c001bd0 con 0x7fe27414be90 2024-03-25T19:50:23.190 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.189+0000 7fe2737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:23.190 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.189+0000 7fe2737fe640 1 -- 172.21.15.81:0/360879712 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe26c001fe0 con 0x7fe27414be90 2024-03-25T19:50:23.190 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.189+0000 7fe27b1fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:23.190 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.189+0000 7fe27b1fb640 4 set_mon_vals no callback set 2024-03-25T19:50:23.190 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.189+0000 7fe2737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:23.190 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.189+0000 7fe2737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:23.191 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.189+0000 7fe2737fe640 10 monclient: dump: 2024-03-25T19:50:23.191 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:23.191 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:23.191 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:23.191 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:23.191 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:23.191 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:23.191 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:23.191 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:23.191 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:23.191 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:23.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.190+0000 7fe27b1fb640 10 monclient: shutdown 2024-03-25T19:50:23.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.190+0000 7fe27b1fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:23.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.190+0000 7fe27b1fb640 1 -- 172.21.15.81:0/360879712 >> v1:172.21.15.98:6789/0 conn(0x7fe27414be90 legacy=0x7fe27414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:23.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.190+0000 7fe27b1fb640 1 -- 172.21.15.81:0/360879712 shutdown_connections 2024-03-25T19:50:23.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.190+0000 7fe27b1fb640 1 -- 172.21.15.81:0/360879712 >> 172.21.15.81:0/360879712 conn(0x7fe2740617a0 msgr2=0x7fe274061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:23.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.190+0000 7fe27b1fb640 1 -- 172.21.15.81:0/360879712 shutdown_connections 2024-03-25T19:50:23.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.190+0000 7fe27b1fb640 1 -- 172.21.15.81:0/360879712 wait complete. 2024-03-25T19:50:23.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.191+0000 7fe27b1fb640 10 monclient: build_initial_monmap 2024-03-25T19:50:23.193 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.191+0000 7fe27b1fb640 10 monclient: monmap: 2024-03-25T19:50:23.194 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:50:23.194 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:23.194 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:50:23.194 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:50:23.194 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:23.194 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:50:23.194 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:50:23.194 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:50:23.194 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:50:23.194 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:23.194 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.191+0000 7fe27b1fb640 1 librados: starting msgr at 2024-03-25T19:50:23.194 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.191+0000 7fe27b1fb640 1 librados: starting objecter 2024-03-25T19:50:23.194 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.191+0000 7fe27b1fb640 1 Processor -- start 2024-03-25T19:50:23.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.191+0000 7fe27b1fb640 1 -- start start 2024-03-25T19:50:23.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.191+0000 7fe27b1fb640 1 librados: setting wanted keys 2024-03-25T19:50:23.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.191+0000 7fe27b1fb640 1 librados: calling monclient init 2024-03-25T19:50:23.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.191+0000 7fe27b1fb640 10 monclient: init 2024-03-25T19:50:23.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.192+0000 7fe27b1fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:23.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.192+0000 7fe27b1fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:23.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.192+0000 7fe27b1fb640 10 monclient(hunting): picked mon.noname-a con 0x7fe2740b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:50:23.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.192+0000 7fe27b1fb640 10 monclient(hunting): picked mon.noname-c con 0x7fe2740b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:50:23.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.192+0000 7fe27b1fb640 10 monclient(hunting): picked mon.noname-b con 0x7fe27414be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:50:23.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.192+0000 7fe27b1fb640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe274140bc0 con 0x7fe2740b1730 2024-03-25T19:50:23.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.192+0000 7fe27b1fb640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe274132620 con 0x7fe27414be90 2024-03-25T19:50:23.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.192+0000 7fe27b1fb640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2741327b0 con 0x7fe2740b4d60 2024-03-25T19:50:23.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.192+0000 7fe27b1fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:23.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.192+0000 7fe27b1fb640 10 monclient(hunting): authenticate will time out at 4458.275353s 2024-03-25T19:50:23.199 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.198+0000 7fe273fff640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fe2740b4d60 0x7fe274131f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:36766/0 (socket says 172.21.15.81:36766) 2024-03-25T19:50:23.199 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.198+0000 7fe273fff640 1 -- 172.21.15.81:0/870663146 learned_addr learned my addr 172.21.15.81:0/870663146 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:50:23.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.200+0000 7fe2717fa640 10 client.?.objecter ms_handle_connect 0x7fe2740b4d60 2024-03-25T19:50:23.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.200+0000 7fe2717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:23.207 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.207+0000 7fe2717fa640 1 -- 172.21.15.81:0/870663146 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992065897 0 0) 0x7fe2741327b0 con 0x7fe2740b4d60 2024-03-25T19:50:23.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.207+0000 7fe2717fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:23.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.207+0000 7fe2717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:23.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.207+0000 7fe2717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:23.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.207+0000 7fe2717fa640 10 monclient(hunting): my global_id is 4586 2024-03-25T19:50:23.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.207+0000 7fe2717fa640 1 -- 172.21.15.81:0/870663146 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe258004eb0 con 0x7fe2740b4d60 2024-03-25T19:50:23.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.212+0000 7fe2717fa640 10 client.?.objecter ms_handle_connect 0x7fe27414be90 2024-03-25T19:50:23.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.212+0000 7fe2717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:23.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.213+0000 7fe2717fa640 1 -- 172.21.15.81:0/870663146 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2683050057 0 0) 0x7fe258004eb0 con 0x7fe2740b4d60 2024-03-25T19:50:23.214 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.213+0000 7fe2717fa640 1 -- 172.21.15.81:0/870663146 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2741327b0 con 0x7fe2740b4d60 2024-03-25T19:50:23.219 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.218+0000 7fe2717fa640 1 -- 172.21.15.81:0/870663146 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe260001b50 con 0x7fe2740b4d60 2024-03-25T19:50:23.219 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.218+0000 7fe2717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:23.219 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.218+0000 7fe2717fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:23.219 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.218+0000 7fe2717fa640 10 monclient(hunting): dump: 2024-03-25T19:50:23.219 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:23.219 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:23.219 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:23.219 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:23.219 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:23.219 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:23.219 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:23.219 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:23.219 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:23.219 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:23.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.218+0000 7fe2717fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:23.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.219+0000 7fe2717fa640 10 client.?.objecter ms_handle_connect 0x7fe2740b1730 2024-03-25T19:50:23.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.219+0000 7fe2717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:23.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe2717fa640 1 -- 172.21.15.81:0/870663146 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1965480665 0 0) 0x7fe2741327b0 con 0x7fe2740b4d60 2024-03-25T19:50:23.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe2717fa640 1 -- 172.21.15.81:0/870663146 >> v1:172.21.15.98:6789/0 conn(0x7fe27414be90 legacy=0x7fe2741daa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:23.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe2717fa640 1 -- 172.21.15.81:0/870663146 >> v1:172.21.15.81:6789/0 conn(0x7fe2740b1730 legacy=0x7fe274140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:23.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe2717fa640 10 monclient: _finish_hunting 0 2024-03-25T19:50:23.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe2717fa640 1 monclient: found mon.c 2024-03-25T19:50:23.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe2717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:23.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe2717fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:50:23.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe2717fa640 1 -- 172.21.15.81:0/870663146 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2741384c0 con 0x7fe2740b4d60 2024-03-25T19:50:23.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe2717fa640 10 monclient: _finish_auth 0 2024-03-25T19:50:23.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe2717fa640 10 monclient: _check_auth_tickets 2024-03-25T19:50:23.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe2717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:23.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe27b1fb640 5 monclient: authenticate success, global_id 4586 2024-03-25T19:50:23.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe27b1fb640 10 monclient: _renew_subs 2024-03-25T19:50:23.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe27b1fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:50:23.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe27b1fb640 1 -- 172.21.15.81:0/870663146 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe274138730 con 0x7fe2740b4d60 2024-03-25T19:50:23.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe27b1fb640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:23.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe27b1fb640 10 monclient: _renew_subs 2024-03-25T19:50:23.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe27b1fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:50:23.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe27b1fb640 1 -- 172.21.15.81:0/870663146 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe274145430 con 0x7fe2740b4d60 2024-03-25T19:50:23.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.228+0000 7fe27b1fb640 1 librados: init done 2024-03-25T19:50:23.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.229+0000 7fe27b1fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:23.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.229+0000 7fe27b1fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:23.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.229+0000 7fe27b1fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:50:23.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.229+0000 7fe27b1fb640 1 -- 172.21.15.81:0/870663146 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2740627b0 con 0x7fe2740b4d60 2024-03-25T19:50:23.527 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.527+0000 7fe2717fa640 1 -- 172.21.15.81:0/870663146 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe260001cb0 con 0x7fe2740b4d60 2024-03-25T19:50:23.528 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.527+0000 7fe2717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:23.528 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.527+0000 7fe2717fa640 1 -- 172.21.15.81:0/870663146 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe260003ae0 con 0x7fe2740b4d60 2024-03-25T19:50:23.528 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.527+0000 7fe2717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:23.528 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.527+0000 7fe2717fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:23.528 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.527+0000 7fe2717fa640 10 monclient: dump: 2024-03-25T19:50:23.528 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:23.528 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:23.528 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:23.528 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:23.528 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:23.528 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:23.528 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:23.528 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:23.529 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:23.529 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:23.529 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.527+0000 7fe272ffd640 4 set_mon_vals no callback set 2024-03-25T19:50:23.529 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.527+0000 7fe2717fa640 1 -- 172.21.15.81:0/870663146 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fe260003d00 con 0x7fe2740b4d60 2024-03-25T19:50:23.529 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.529+0000 7fe278f97640 10 monclient: get_auth_request con 0x7fe25806dc60 auth_method 0 2024-03-25T19:50:23.529 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.529+0000 7fe2717fa640 10 client.4586.objecter ms_handle_connect 0x7fe25806dc60 2024-03-25T19:50:23.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.535+0000 7fe2717fa640 1 -- 172.21.15.81:0/870663146 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fe2600549d0 con 0x7fe2740b4d60 2024-03-25T19:50:23.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.535+0000 7fe2717fa640 10 client.4586.objecter ms_dispatch 0x7fe274000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:23.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.535+0000 7fe2717fa640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:23.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.535+0000 7fe2717fa640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:23.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.535+0000 7fe2717fa640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-25T19:50:23.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.535+0000 7fe2717fa640 1 -- 172.21.15.81:0/870663146 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe260050310 con 0x7fe2740b4d60 2024-03-25T19:50:23.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.535+0000 7fe2717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:23.535 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.535+0000 7fe2717fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:23.638 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:08:46.280 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:23.653 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:08:46.295 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:23.689 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:08:46.331 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:23.693 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:08:46.335 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:23.702 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:08:46.336 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:23.705 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:08:46.347 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:23.710 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:08:46.345 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:23.726 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:08:46.367 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:23.731 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:08:46.365 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:23.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.758+0000 7fe27b1fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:23.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.758+0000 7fe27b1fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:23.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.758+0000 7fe27b1fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:50:23.757 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.758+0000 7fe27b1fb640 1 -- 172.21.15.81:0/870663146 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe274140d70 con 0x7fe2740b4d60 2024-03-25T19:50:23.869 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.869+0000 7fe2717fa640 1 -- 172.21.15.81:0/870663146 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fe274140d70 con 0x7fe2740b4d60 2024-03-25T19:50:23.869 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.869+0000 7fe2717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:23.869 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.869+0000 7fe2717fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:23.869 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:50:23.869 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:23.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.871+0000 7fe27b1fb640 10 librados: watch_flush enter 2024-03-25T19:50:23.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.871+0000 7fe27b1fb640 10 librados: watch_flush exit 2024-03-25T19:50:23.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.871+0000 7fe27b1fb640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-25T19:50:23.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.871+0000 7fe27b1fb640 10 client.4586.objecter successfully canceled tick 2024-03-25T19:50:23.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.871+0000 7fe27b1fb640 1 -- 172.21.15.81:0/870663146 >> v1:172.21.15.98:6800/3980829775 conn(0x7fe25806dc60 legacy=0x7fe258070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:23.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.871+0000 7fe27b1fb640 10 monclient: shutdown 2024-03-25T19:50:23.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.871+0000 7fe27b1fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:23.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.871+0000 7fe27b1fb640 1 -- 172.21.15.81:0/870663146 >> v1:172.21.15.81:6790/0 conn(0x7fe2740b4d60 legacy=0x7fe274131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:23.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.872+0000 7fe27b1fb640 1 -- 172.21.15.81:0/870663146 shutdown_connections 2024-03-25T19:50:23.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.872+0000 7fe27b1fb640 1 -- 172.21.15.81:0/870663146 >> 172.21.15.81:0/870663146 conn(0x7fe2740617a0 msgr2=0x7fe2740b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:23.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.872+0000 7fe27b1fb640 1 -- 172.21.15.81:0/870663146 shutdown_connections 2024-03-25T19:50:23.873 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.872+0000 7fe27b1fb640 1 -- 172.21.15.81:0/870663146 wait complete. 2024-03-25T19:50:23.873 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:23.872+0000 7fe27b1fb640 1 librados: shutdown 2024-03-25T19:50:23.882 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:30.689 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:30.806 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.807+0000 7fbbba6f1640 10 monclient: get_monmap_and_config 2024-03-25T19:50:30.806 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.807+0000 7fbbba6f1640 10 monclient: build_initial_monmap 2024-03-25T19:50:30.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.807+0000 7fbbba6f1640 10 monclient: monmap: 2024-03-25T19:50:30.807 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:50:30.807 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:30.807 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:50:30.808023+0000 2024-03-25T19:50:30.807 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:50:30.808023+0000 2024-03-25T19:50:30.807 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:30.807 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:30.807 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:50:30.807 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:50:30.807 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:50:30.807 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:30.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.807+0000 7fbbba6f1640 1 Processor -- start 2024-03-25T19:50:30.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.808+0000 7fbbba6f1640 1 -- start start 2024-03-25T19:50:30.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.808+0000 7fbbba6f1640 10 monclient: init 2024-03-25T19:50:30.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.808+0000 7fbbba6f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:30.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.808+0000 7fbbba6f1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:30.809 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.808+0000 7fbbba6f1640 10 monclient(hunting): picked mon.noname-b con 0x7fbbb40a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:50:30.809 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.808+0000 7fbbba6f1640 10 monclient(hunting): picked mon.noname-a con 0x7fbbb413f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:50:30.809 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.808+0000 7fbbba6f1640 10 monclient(hunting): picked mon.noname-c con 0x7fbbb41430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:50:30.809 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.808+0000 7fbbba6f1640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb4142520 con 0x7fbbb413f710 2024-03-25T19:50:30.809 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.808+0000 7fbbba6f1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb40b28a0 con 0x7fbbb41430c0 2024-03-25T19:50:30.809 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.808+0000 7fbbba6f1640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb40b2a30 con 0x7fbbb40a98f0 2024-03-25T19:50:30.809 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.808+0000 7fbbba6f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:30.809 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.808+0000 7fbbba6f1640 10 monclient(hunting): authenticate will time out at 4465.890476s 2024-03-25T19:50:30.813 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.813+0000 7fbbb8c8e640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbbb41430c0 0x7fbbb414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:33432/0 (socket says 172.21.15.81:33432) 2024-03-25T19:50:30.813 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.813+0000 7fbbb8c8e640 1 -- 172.21.15.81:0/2296393587 learned_addr learned my addr 172.21.15.81:0/2296393587 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:50:30.828 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.828+0000 7fbbb2ffd640 1 -- 172.21.15.81:0/2296393587 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522842270 0 0) 0x7fbbb40b28a0 con 0x7fbbb41430c0 2024-03-25T19:50:30.828 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.828+0000 7fbbb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:30.828 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.828+0000 7fbbb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:30.828 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.828+0000 7fbbb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:30.828 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.828+0000 7fbbb2ffd640 10 monclient(hunting): my global_id is 4603 2024-03-25T19:50:30.828 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.828+0000 7fbbb2ffd640 1 -- 172.21.15.81:0/2296393587 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb9c003520 con 0x7fbbb41430c0 2024-03-25T19:50:30.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.840+0000 7fbbb2ffd640 1 -- 172.21.15.81:0/2296393587 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862732900 0 0) 0x7fbbb40b2a30 con 0x7fbbb40a98f0 2024-03-25T19:50:30.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.840+0000 7fbbb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:30.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.840+0000 7fbbb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:30.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.840+0000 7fbbb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:30.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.840+0000 7fbbb2ffd640 10 monclient(hunting): my global_id is 4589 2024-03-25T19:50:30.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.840+0000 7fbbb2ffd640 1 -- 172.21.15.81:0/2296393587 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbb40b28a0 con 0x7fbbb40a98f0 2024-03-25T19:50:30.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.845+0000 7fbbb2ffd640 1 -- 172.21.15.81:0/2296393587 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505236943 0 0) 0x7fbbb4142520 con 0x7fbbb413f710 2024-03-25T19:50:30.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.845+0000 7fbbb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:30.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.845+0000 7fbbb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:30.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.845+0000 7fbbb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:30.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.845+0000 7fbbb2ffd640 10 monclient(hunting): my global_id is 4617 2024-03-25T19:50:30.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.845+0000 7fbbb2ffd640 1 -- 172.21.15.81:0/2296393587 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbb40b2a30 con 0x7fbbb413f710 2024-03-25T19:50:30.848 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.848+0000 7fbbb2ffd640 1 -- 172.21.15.81:0/2296393587 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 402993804 0 0) 0x7fbb9c003520 con 0x7fbbb41430c0 2024-03-25T19:50:30.848 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.848+0000 7fbbb2ffd640 1 -- 172.21.15.81:0/2296393587 >> v1:172.21.15.81:6790/0 conn(0x7fbbb40a98f0 legacy=0x7fbbb40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:30.848 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.848+0000 7fbbb2ffd640 1 -- 172.21.15.81:0/2296393587 >> v1:172.21.15.81:6789/0 conn(0x7fbbb413f710 legacy=0x7fbbb41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:30.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.848+0000 7fbbb2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:30.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.848+0000 7fbbb2ffd640 1 monclient: found mon.noname-c 2024-03-25T19:50:30.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.848+0000 7fbbb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:30.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.848+0000 7fbbb2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:50:30.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.848+0000 7fbbb2ffd640 1 -- 172.21.15.81:0/2296393587 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb40b44c0 con 0x7fbbb41430c0 2024-03-25T19:50:30.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.848+0000 7fbbb2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:30.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.848+0000 7fbbb2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:30.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.848+0000 7fbbb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:30.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.848+0000 7fbbb2ffd640 1 -- 172.21.15.81:0/2296393587 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fbbac001a10 con 0x7fbbb41430c0 2024-03-25T19:50:30.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.848+0000 7fbbb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:30.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.848+0000 7fbbb2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:30.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.848+0000 7fbbb2ffd640 10 monclient: dump: 2024-03-25T19:50:30.849 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:30.849 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:30.849 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:30.850 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:30.850 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:30.850 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:30.850 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:30.850 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:30.850 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:30.850 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:30.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.848+0000 7fbbba6f1640 5 monclient: authenticate success, global_id 4603 2024-03-25T19:50:30.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.848+0000 7fbbba6f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:30.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.860+0000 7fbbb2ffd640 1 -- 172.21.15.81:0/2296393587 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbac001b70 con 0x7fbbb41430c0 2024-03-25T19:50:30.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.860+0000 7fbbb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:30.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.860+0000 7fbbb2ffd640 1 -- 172.21.15.81:0/2296393587 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fbbac001f80 con 0x7fbbb41430c0 2024-03-25T19:50:30.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.860+0000 7fbbba6f1640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:30.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.860+0000 7fbbba6f1640 4 set_mon_vals no callback set 2024-03-25T19:50:30.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.860+0000 7fbbb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:30.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.860+0000 7fbbb2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:30.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.860+0000 7fbbb2ffd640 10 monclient: dump: 2024-03-25T19:50:30.861 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:30.861 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:30.861 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:30.861 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:30.861 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:30.861 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:30.861 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:30.861 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:30.861 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:30.861 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:30.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.861+0000 7fbbba6f1640 10 monclient: shutdown 2024-03-25T19:50:30.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.861+0000 7fbbba6f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:30.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.861+0000 7fbbba6f1640 1 -- 172.21.15.81:0/2296393587 >> v1:172.21.15.98:6789/0 conn(0x7fbbb41430c0 legacy=0x7fbbb414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:30.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.861+0000 7fbbba6f1640 1 -- 172.21.15.81:0/2296393587 shutdown_connections 2024-03-25T19:50:30.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.861+0000 7fbbba6f1640 1 -- 172.21.15.81:0/2296393587 >> 172.21.15.81:0/2296393587 conn(0x7fbbb40b7ea0 msgr2=0x7fbbb40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:30.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.861+0000 7fbbba6f1640 1 -- 172.21.15.81:0/2296393587 shutdown_connections 2024-03-25T19:50:30.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.861+0000 7fbbba6f1640 1 -- 172.21.15.81:0/2296393587 wait complete. 2024-03-25T19:50:30.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.862+0000 7fbbba6f1640 10 monclient: build_initial_monmap 2024-03-25T19:50:30.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.862+0000 7fbbba6f1640 10 monclient: monmap: 2024-03-25T19:50:30.863 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:50:30.863 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:30.863 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:50:30.863 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:50:30.863 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:30.864 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:50:30.864 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:50:30.864 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:50:30.864 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:50:30.864 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:30.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.863+0000 7fbbba6f1640 1 librados: starting msgr at 2024-03-25T19:50:30.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.863+0000 7fbbba6f1640 1 librados: starting objecter 2024-03-25T19:50:30.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.863+0000 7fbbba6f1640 1 Processor -- start 2024-03-25T19:50:30.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.863+0000 7fbbba6f1640 1 -- start start 2024-03-25T19:50:30.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.863+0000 7fbbba6f1640 1 librados: setting wanted keys 2024-03-25T19:50:30.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.863+0000 7fbbba6f1640 1 librados: calling monclient init 2024-03-25T19:50:30.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.863+0000 7fbbba6f1640 10 monclient: init 2024-03-25T19:50:30.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.864+0000 7fbbba6f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:30.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.864+0000 7fbbba6f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:30.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.864+0000 7fbbba6f1640 10 monclient(hunting): picked mon.noname-b con 0x7fbbb40a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:50:30.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.864+0000 7fbbba6f1640 10 monclient(hunting): picked mon.noname-c con 0x7fbbb413f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:50:30.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.864+0000 7fbbba6f1640 10 monclient(hunting): picked mon.noname-a con 0x7fbbb41430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:50:30.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.864+0000 7fbbba6f1640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb4135230 con 0x7fbbb41430c0 2024-03-25T19:50:30.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.864+0000 7fbbba6f1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb4135da0 con 0x7fbbb40a98f0 2024-03-25T19:50:30.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.864+0000 7fbbba6f1640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb4135f30 con 0x7fbbb413f710 2024-03-25T19:50:30.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.864+0000 7fbbba6f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:30.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.864+0000 7fbbba6f1640 10 monclient(hunting): authenticate will time out at 4465.946187s 2024-03-25T19:50:30.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.868+0000 7fbbb3fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fbbb40a98f0 0x7fbbb41d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:33468/0 (socket says 172.21.15.81:33468) 2024-03-25T19:50:30.868 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.868+0000 7fbbb3fff640 1 -- 172.21.15.81:0/1986265832 learned_addr learned my addr 172.21.15.81:0/1986265832 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:50:30.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.871+0000 7fbbb0ff9640 10 client.?.objecter ms_handle_connect 0x7fbbb40a98f0 2024-03-25T19:50:30.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.871+0000 7fbbb0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:30.871 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.872+0000 7fbbb0ff9640 10 client.?.objecter ms_handle_connect 0x7fbbb41430c0 2024-03-25T19:50:30.871 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.872+0000 7fbbb0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:30.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.877+0000 7fbbb37fe640 1 -- 172.21.15.81:0/1986265832 >> v1:172.21.15.81:6790/0 conn(0x7fbbb413f710 legacy=0x7fbbb41dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T19:50:30.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.877+0000 7fbbb37fe640 1 -- 172.21.15.81:0/1986265832 >> v1:172.21.15.81:6790/0 conn(0x7fbbb413f710 legacy=0x7fbbb41dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:50:30.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.877+0000 7fbbb37fe640 1 --1- 172.21.15.81:0/1986265832 >> v1:172.21.15.81:6790/0 conn(0x7fbbb413f710 0x7fbbb41dae50 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-25T19:50:30.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.877+0000 7fbbb37fe640 1 --1- 172.21.15.81:0/1986265832 >> v1:172.21.15.81:6790/0 conn(0x7fbbb413f710 0x7fbbb41dae50 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T19:50:30.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.877+0000 7fbbb0ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.81:6790/0 2024-03-25T19:50:30.878 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.878+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337130710 0 0) 0x7fbbb4135230 con 0x7fbbb41430c0 2024-03-25T19:50:30.878 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.878+0000 7fbbb0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:30.878 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.878+0000 7fbbb0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:30.878 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.878+0000 7fbbb0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:30.878 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.878+0000 7fbbb0ff9640 10 monclient(hunting): my global_id is 4620 2024-03-25T19:50:30.878 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.878+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbba0004eb0 con 0x7fbbb41430c0 2024-03-25T19:50:30.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.880+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30688371 0 0) 0x7fbbb4135da0 con 0x7fbbb40a98f0 2024-03-25T19:50:30.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.880+0000 7fbbb0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:30.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.880+0000 7fbbb0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:30.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.880+0000 7fbbb0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:30.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.880+0000 7fbbb0ff9640 10 monclient(hunting): my global_id is 4606 2024-03-25T19:50:30.880 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.880+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbb4135230 con 0x7fbbb40a98f0 2024-03-25T19:50:30.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.893+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3678479539 0 0) 0x7fbbb4135230 con 0x7fbbb40a98f0 2024-03-25T19:50:30.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.893+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbb4135da0 con 0x7fbbb40a98f0 2024-03-25T19:50:30.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.893+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fbba40019b0 con 0x7fbbb40a98f0 2024-03-25T19:50:30.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.893+0000 7fbbb0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:30.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.893+0000 7fbbb0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:30.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.893+0000 7fbbb0ff9640 10 monclient(hunting): dump: 2024-03-25T19:50:30.893 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:30.893 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:30.893 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:30.893 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:30.893 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:30.893 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:30.893 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:30.894 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:30.894 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:30.894 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:30.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.893+0000 7fbbb0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:30.901 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.901+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4051139914 0 0) 0x7fbba0004eb0 con 0x7fbbb41430c0 2024-03-25T19:50:30.901 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.901+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbb4135230 con 0x7fbbb41430c0 2024-03-25T19:50:30.901 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.901+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fbbac001e80 con 0x7fbbb41430c0 2024-03-25T19:50:30.901 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.901+0000 7fbbb0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:30.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.901+0000 7fbbb0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:30.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.901+0000 7fbbb0ff9640 10 monclient(hunting): dump: 2024-03-25T19:50:30.902 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:30.902 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:30.902 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:30.902 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:30.902 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:30.902 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:30.902 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:30.902 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:30.902 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:30.902 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:30.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 968518286 0 0) 0x7fbbb4135da0 con 0x7fbbb40a98f0 2024-03-25T19:50:30.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 >> v1:172.21.15.81:6790/0 conn(0x7fbbb413f710 legacy=0x7fbbb41dae50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:50:30.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 >> v1:172.21.15.81:6789/0 conn(0x7fbbb41430c0 legacy=0x7fbbb41db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:30.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbb0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:50:30.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbb0ff9640 1 monclient: found mon.b 2024-03-25T19:50:30.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbb0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:30.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbb0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:30.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb41360c0 con 0x7fbbb40a98f0 2024-03-25T19:50:30.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbb0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:50:30.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbb0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:50:30.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbb0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:30.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbba6f1640 5 monclient: authenticate success, global_id 4606 2024-03-25T19:50:30.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbba6f1640 10 monclient: _renew_subs 2024-03-25T19:50:30.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbba6f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:30.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbba6f1640 1 -- 172.21.15.81:0/1986265832 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb41d5000 con 0x7fbbb40a98f0 2024-03-25T19:50:30.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbba6f1640 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:30.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbba6f1640 10 monclient: _renew_subs 2024-03-25T19:50:30.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbba6f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:30.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbba6f1640 1 -- 172.21.15.81:0/1986265832 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb41d41e0 con 0x7fbbb40a98f0 2024-03-25T19:50:30.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.911+0000 7fbbba6f1640 1 librados: init done 2024-03-25T19:50:30.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.912+0000 7fbbba6f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:30.913 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.912+0000 7fbbba6f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:30.913 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.912+0000 7fbbba6f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:30.913 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:30.912+0000 7fbbba6f1640 1 -- 172.21.15.81:0/1986265832 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb4135090 con 0x7fbbb40a98f0 2024-03-25T19:50:31.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.208+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbba4003610 con 0x7fbbb40a98f0 2024-03-25T19:50:31.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.208+0000 7fbbb0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:31.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.208+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fbba4003880 con 0x7fbbb40a98f0 2024-03-25T19:50:31.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.208+0000 7fbbb0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:31.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.208+0000 7fbbb0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:31.208 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.208+0000 7fbbb0ff9640 10 monclient: dump: 2024-03-25T19:50:31.208 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:31.208 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:31.208 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:31.209 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:31.209 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:31.209 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:31.209 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:31.209 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:31.209 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:31.209 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:31.209 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.208+0000 7fbbb27fc640 4 set_mon_vals no callback set 2024-03-25T19:50:31.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.209+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fbba4003aa0 con 0x7fbbb40a98f0 2024-03-25T19:50:31.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.210+0000 7fbbb37fe640 10 monclient: get_auth_request con 0x7fbba006de90 auth_method 0 2024-03-25T19:50:31.210 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.210+0000 7fbbb0ff9640 10 client.4606.objecter ms_handle_connect 0x7fbba006de90 2024-03-25T19:50:31.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.213+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fbba4054630 con 0x7fbbb40a98f0 2024-03-25T19:50:31.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.213+0000 7fbbb0ff9640 10 client.4606.objecter ms_dispatch 0x7fbbb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:31.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.213+0000 7fbbb0ff9640 3 client.4606.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:31.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.213+0000 7fbbb0ff9640 3 client.4606.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:31.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.213+0000 7fbbb0ff9640 20 client.4606.objecter dump_active .. 0 homeless 2024-03-25T19:50:31.214 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.214+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbba4050140 con 0x7fbbb40a98f0 2024-03-25T19:50:31.214 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.214+0000 7fbbb0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:31.214 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.214+0000 7fbbb0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:31.466 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.466+0000 7fbbba6f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:31.466 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.466+0000 7fbbba6f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:31.466 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.466+0000 7fbbba6f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:31.466 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.466+0000 7fbbba6f1640 1 -- 172.21.15.81:0/1986265832 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbbb413b700 con 0x7fbbb40a98f0 2024-03-25T19:50:31.492 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.493+0000 7fbbb0ff9640 1 -- 172.21.15.81:0/1986265832 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fbba4052070 con 0x7fbbb40a98f0 2024-03-25T19:50:31.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.493+0000 7fbbb0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:31.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.493+0000 7fbbb0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:31.493 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:50:31.493 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:31.495 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.495+0000 7fbbba6f1640 10 librados: watch_flush enter 2024-03-25T19:50:31.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.495+0000 7fbbba6f1640 10 librados: watch_flush exit 2024-03-25T19:50:31.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.495+0000 7fbbba6f1640 20 client.4606.objecter shutdown clearing up homeless session... 2024-03-25T19:50:31.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.495+0000 7fbbba6f1640 10 client.4606.objecter successfully canceled tick 2024-03-25T19:50:31.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.495+0000 7fbbba6f1640 1 -- 172.21.15.81:0/1986265832 >> v1:172.21.15.98:6800/3980829775 conn(0x7fbba006de90 legacy=0x7fbba0070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:31.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.495+0000 7fbbba6f1640 10 monclient: shutdown 2024-03-25T19:50:31.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.495+0000 7fbbba6f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:31.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.495+0000 7fbbba6f1640 1 -- 172.21.15.81:0/1986265832 >> v1:172.21.15.98:6789/0 conn(0x7fbbb40a98f0 legacy=0x7fbbb41d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:31.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.496+0000 7fbbba6f1640 1 -- 172.21.15.81:0/1986265832 shutdown_connections 2024-03-25T19:50:31.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.496+0000 7fbbba6f1640 1 -- 172.21.15.81:0/1986265832 >> 172.21.15.81:0/1986265832 conn(0x7fbbb40b7ea0 msgr2=0x7fbbb414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:31.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.496+0000 7fbbba6f1640 1 -- 172.21.15.81:0/1986265832 shutdown_connections 2024-03-25T19:50:31.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.496+0000 7fbbba6f1640 1 -- 172.21.15.81:0/1986265832 wait complete. 2024-03-25T19:50:31.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:31.496+0000 7fbbba6f1640 1 librados: shutdown 2024-03-25T19:50:31.508 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:38.314 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:38.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.429+0000 7ff00dcd6640 10 monclient: get_monmap_and_config 2024-03-25T19:50:38.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.429+0000 7ff00dcd6640 10 monclient: build_initial_monmap 2024-03-25T19:50:38.428 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.429+0000 7ff00dcd6640 10 monclient: monmap: 2024-03-25T19:50:38.429 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:50:38.429 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:38.429 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:50:38.430056+0000 2024-03-25T19:50:38.429 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:50:38.430056+0000 2024-03-25T19:50:38.429 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:38.429 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:38.429 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:50:38.429 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:50:38.429 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:50:38.429 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:38.430 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.429+0000 7ff00dcd6640 1 Processor -- start 2024-03-25T19:50:38.430 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.430+0000 7ff00dcd6640 1 -- start start 2024-03-25T19:50:38.430 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.430+0000 7ff00dcd6640 10 monclient: init 2024-03-25T19:50:38.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.430+0000 7ff00dcd6640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:38.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.430+0000 7ff00dcd6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:38.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.430+0000 7ff00dcd6640 10 monclient(hunting): picked mon.noname-c con 0x7ff0080b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:50:38.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.430+0000 7ff00dcd6640 10 monclient(hunting): picked mon.noname-b con 0x7ff0080b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:50:38.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.430+0000 7ff00dcd6640 10 monclient(hunting): picked mon.noname-a con 0x7ff00814be90 addr v1:172.21.15.81:6789/0 2024-03-25T19:50:38.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.430+0000 7ff00dcd6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0080af940 con 0x7ff00814be90 2024-03-25T19:50:38.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.430+0000 7ff00dcd6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0080b0a70 con 0x7ff0080b4d60 2024-03-25T19:50:38.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.430+0000 7ff00dcd6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0080b0d50 con 0x7ff0080b1730 2024-03-25T19:50:38.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.430+0000 7ff00dcd6640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:38.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.430+0000 7ff00dcd6640 10 monclient(hunting): authenticate will time out at 4473.512769s 2024-03-25T19:50:38.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.436+0000 7ff007fff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7ff00814be90 0x7ff00814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:43776/0 (socket says 172.21.15.81:43776) 2024-03-25T19:50:38.436 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.436+0000 7ff007fff640 1 -- 172.21.15.81:0/4073505458 learned_addr learned my addr 172.21.15.81:0/4073505458 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:50:38.452 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.452+0000 7ff0067fc640 1 -- 172.21.15.81:0/4073505458 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230969604 0 0) 0x7ff0080b0d50 con 0x7ff0080b1730 2024-03-25T19:50:38.452 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.452+0000 7ff0067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:38.452 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.452+0000 7ff0067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:38.452 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.452+0000 7ff0067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:38.452 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.452+0000 7ff0067fc640 10 monclient(hunting): my global_id is 4592 2024-03-25T19:50:38.452 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.452+0000 7ff0067fc640 1 -- 172.21.15.81:0/4073505458 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feff0003520 con 0x7ff0080b1730 2024-03-25T19:50:38.461 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 1 -- 172.21.15.81:0/4073505458 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627079291 0 0) 0x7ff0080b0a70 con 0x7ff0080b4d60 2024-03-25T19:50:38.461 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:38.461 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:38.461 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:38.461 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:50:38.462 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 1 -- 172.21.15.81:0/4073505458 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0080b0d50 con 0x7ff0080b4d60 2024-03-25T19:50:38.462 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 1 -- 172.21.15.81:0/4073505458 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3269176947 0 0) 0x7feff0003520 con 0x7ff0080b1730 2024-03-25T19:50:38.462 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 1 -- 172.21.15.81:0/4073505458 >> v1:172.21.15.98:6789/0 conn(0x7ff0080b4d60 legacy=0x7ff00813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:38.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 1 -- 172.21.15.81:0/4073505458 >> v1:172.21.15.81:6789/0 conn(0x7ff00814be90 legacy=0x7ff00814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:38.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:38.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 1 monclient: found mon.noname-b 2024-03-25T19:50:38.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:38.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:50:38.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 1 -- 172.21.15.81:0/4073505458 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0080b1030 con 0x7ff0080b1730 2024-03-25T19:50:38.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:38.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:38.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:38.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 1 -- 172.21.15.81:0/4073505458 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7feffc001690 con 0x7ff0080b1730 2024-03-25T19:50:38.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:38.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:38.463 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff0067fc640 10 monclient: dump: 2024-03-25T19:50:38.463 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:38.463 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:38.463 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:38.464 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:38.464 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:38.464 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:38.464 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:38.464 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:38.464 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:38.464 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:38.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff00dcd6640 5 monclient: authenticate success, global_id 4592 2024-03-25T19:50:38.464 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.462+0000 7ff00dcd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:38.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.479+0000 7ff0067fc640 1 -- 172.21.15.81:0/4073505458 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feffc0017f0 con 0x7ff0080b1730 2024-03-25T19:50:38.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.479+0000 7ff0067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:38.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.479+0000 7ff0067fc640 1 -- 172.21.15.81:0/4073505458 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7feffc001c00 con 0x7ff0080b1730 2024-03-25T19:50:38.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.479+0000 7ff00dcd6640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:38.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.479+0000 7ff00dcd6640 4 set_mon_vals no callback set 2024-03-25T19:50:38.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.479+0000 7ff0067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:38.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.479+0000 7ff0067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:38.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.479+0000 7ff0067fc640 10 monclient: dump: 2024-03-25T19:50:38.479 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:38.479 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:38.479 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:38.479 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:38.479 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:38.479 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:38.479 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:38.479 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:38.479 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:38.479 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:38.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.479+0000 7ff00dcd6640 10 monclient: shutdown 2024-03-25T19:50:38.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.479+0000 7ff00dcd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:38.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.479+0000 7ff00dcd6640 1 -- 172.21.15.81:0/4073505458 >> v1:172.21.15.81:6790/0 conn(0x7ff0080b1730 legacy=0x7ff0080b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:38.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.479+0000 7ff00dcd6640 1 -- 172.21.15.81:0/4073505458 shutdown_connections 2024-03-25T19:50:38.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.480+0000 7ff00dcd6640 1 -- 172.21.15.81:0/4073505458 >> 172.21.15.81:0/4073505458 conn(0x7ff0080617a0 msgr2=0x7ff008061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:38.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.480+0000 7ff00dcd6640 1 -- 172.21.15.81:0/4073505458 shutdown_connections 2024-03-25T19:50:38.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.480+0000 7ff00dcd6640 1 -- 172.21.15.81:0/4073505458 wait complete. 2024-03-25T19:50:38.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.480+0000 7ff00dcd6640 10 monclient: build_initial_monmap 2024-03-25T19:50:38.481 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.480+0000 7ff00dcd6640 10 monclient: monmap: 2024-03-25T19:50:38.481 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:50:38.481 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:38.482 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:50:38.482 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:50:38.482 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:38.482 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:50:38.482 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:50:38.482 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:50:38.482 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:50:38.482 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:38.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.481+0000 7ff00dcd6640 1 librados: starting msgr at 2024-03-25T19:50:38.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.481+0000 7ff00dcd6640 1 librados: starting objecter 2024-03-25T19:50:38.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.481+0000 7ff00dcd6640 1 Processor -- start 2024-03-25T19:50:38.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.481+0000 7ff00dcd6640 1 -- start start 2024-03-25T19:50:38.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.481+0000 7ff00dcd6640 1 librados: setting wanted keys 2024-03-25T19:50:38.482 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.481+0000 7ff00dcd6640 1 librados: calling monclient init 2024-03-25T19:50:38.483 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.481+0000 7ff00dcd6640 10 monclient: init 2024-03-25T19:50:38.483 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.482+0000 7ff00dcd6640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:38.484 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.482+0000 7ff00dcd6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:38.484 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.482+0000 7ff00dcd6640 10 monclient(hunting): picked mon.noname-c con 0x7ff0080b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:50:38.484 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.482+0000 7ff00dcd6640 10 monclient(hunting): picked mon.noname-a con 0x7ff0080b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:50:38.484 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.482+0000 7ff00dcd6640 10 monclient(hunting): picked mon.noname-b con 0x7ff00814be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:50:38.484 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.482+0000 7ff00dcd6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff008136250 con 0x7ff0080b4d60 2024-03-25T19:50:38.484 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.482+0000 7ff00dcd6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff008136af0 con 0x7ff00814be90 2024-03-25T19:50:38.484 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.482+0000 7ff00dcd6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff008136c80 con 0x7ff0080b1730 2024-03-25T19:50:38.484 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.482+0000 7ff00dcd6640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:38.484 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.482+0000 7ff00dcd6640 10 monclient(hunting): authenticate will time out at 4473.564268s 2024-03-25T19:50:38.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.490+0000 7ff007fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7ff00814be90 0x7ff0081e2f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:56918/0 (socket says 172.21.15.81:56918) 2024-03-25T19:50:38.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.490+0000 7ff007fff640 1 -- 172.21.15.81:0/2218125345 learned_addr learned my addr 172.21.15.81:0/2218125345 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:50:38.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.491+0000 7fefeffff640 10 client.?.objecter ms_handle_connect 0x7ff00814be90 2024-03-25T19:50:38.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.491+0000 7fefeffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:38.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.494+0000 7fefeffff640 10 client.?.objecter ms_handle_connect 0x7ff0080b4d60 2024-03-25T19:50:38.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.494+0000 7fefeffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:38.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.498+0000 7fefeffff640 10 client.?.objecter ms_handle_connect 0x7ff0080b1730 2024-03-25T19:50:38.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.498+0000 7fefeffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:38.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.504+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941210190 0 0) 0x7ff008136af0 con 0x7ff00814be90 2024-03-25T19:50:38.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.504+0000 7fefeffff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:38.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.504+0000 7fefeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:38.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.504+0000 7fefeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:38.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.504+0000 7fefeffff640 10 monclient(hunting): my global_id is 4612 2024-03-25T19:50:38.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.504+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feff4004eb0 con 0x7ff00814be90 2024-03-25T19:50:38.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.513+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503790926 0 0) 0x7ff008136250 con 0x7ff0080b4d60 2024-03-25T19:50:38.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.513+0000 7fefeffff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:38.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.513+0000 7fefeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:38.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.513+0000 7fefeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:38.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.513+0000 7fefeffff640 10 monclient(hunting): my global_id is 4626 2024-03-25T19:50:38.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.513+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff008136af0 con 0x7ff0080b4d60 2024-03-25T19:50:38.518 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.519+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3147436592 0 0) 0x7feff4004eb0 con 0x7ff00814be90 2024-03-25T19:50:38.518 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.519+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff008136250 con 0x7ff00814be90 2024-03-25T19:50:38.519 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.519+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7ff000001b10 con 0x7ff00814be90 2024-03-25T19:50:38.519 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.519+0000 7fefeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:38.519 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.519+0000 7fefeffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:38.519 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.519+0000 7fefeffff640 10 monclient(hunting): dump: 2024-03-25T19:50:38.519 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:38.519 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:38.519 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:38.519 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:38.519 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:38.519 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:38.519 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:38.519 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:38.519 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:38.520 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:38.520 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.519+0000 7fefeffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:38.524 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.524+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 467178937 0 0) 0x7ff008136c80 con 0x7ff0080b1730 2024-03-25T19:50:38.524 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.524+0000 7fefeffff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:38.524 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.524+0000 7fefeffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:38.524 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.524+0000 7fefeffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:38.524 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.524+0000 7fefeffff640 10 monclient(hunting): my global_id is 4595 2024-03-25T19:50:38.524 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.524+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feff4004eb0 con 0x7ff0080b1730 2024-03-25T19:50:38.529 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.529+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3816579805 0 0) 0x7ff008136af0 con 0x7ff0080b4d60 2024-03-25T19:50:38.529 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.529+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff008136c80 con 0x7ff0080b4d60 2024-03-25T19:50:38.529 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.529+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7feffc001b40 con 0x7ff0080b4d60 2024-03-25T19:50:38.529 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.529+0000 7fefeffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:38.529 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.529+0000 7fefeffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:38.529 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.529+0000 7fefeffff640 10 monclient(hunting): dump: 2024-03-25T19:50:38.529 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:38.529 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:38.529 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:38.529 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:38.529 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:38.529 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:38.530 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:38.530 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:38.530 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:38.530 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:38.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3976359000 0 0) 0x7ff008136250 con 0x7ff00814be90 2024-03-25T19:50:38.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 >> v1:172.21.15.81:6790/0 conn(0x7ff0080b1730 legacy=0x7ff0081d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:38.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 >> v1:172.21.15.81:6789/0 conn(0x7ff0080b4d60 legacy=0x7ff00812fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:38.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7fefeffff640 10 monclient: _finish_hunting 0 2024-03-25T19:50:38.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7fefeffff640 1 monclient: found mon.b 2024-03-25T19:50:38.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7fefeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:38.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7fefeffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:38.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff008138210 con 0x7ff00814be90 2024-03-25T19:50:38.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7fefeffff640 10 monclient: _finish_auth 0 2024-03-25T19:50:38.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7fefeffff640 10 monclient: _check_auth_tickets 2024-03-25T19:50:38.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7fefeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:38.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7ff00dcd6640 5 monclient: authenticate success, global_id 4612 2024-03-25T19:50:38.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7ff00dcd6640 10 monclient: _renew_subs 2024-03-25T19:50:38.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7ff00dcd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:38.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7ff00dcd6640 1 -- 172.21.15.81:0/2218125345 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0081384c0 con 0x7ff00814be90 2024-03-25T19:50:38.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7ff00dcd6640 10 client.4612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:38.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7ff00dcd6640 10 monclient: _renew_subs 2024-03-25T19:50:38.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7ff00dcd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:38.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7ff00dcd6640 1 -- 172.21.15.81:0/2218125345 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff008145700 con 0x7ff00814be90 2024-03-25T19:50:38.540 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.539+0000 7ff00dcd6640 1 librados: init done 2024-03-25T19:50:38.541 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.540+0000 7ff00dcd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:38.541 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.540+0000 7ff00dcd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:38.541 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.540+0000 7ff00dcd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:38.541 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.540+0000 7ff00dcd6640 1 -- 172.21.15.81:0/2218125345 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0080627b0 con 0x7ff00814be90 2024-03-25T19:50:38.750 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.750+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff000002000 con 0x7ff00814be90 2024-03-25T19:50:38.750 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.750+0000 7fefeffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:38.750 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.750+0000 7ff005ffb640 4 set_mon_vals no callback set 2024-03-25T19:50:38.811 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.812+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7ff000004d00 con 0x7ff00814be90 2024-03-25T19:50:38.811 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.812+0000 7fefeffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:38.812 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.812+0000 7fefeffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:38.812 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.812+0000 7fefeffff640 10 monclient: dump: 2024-03-25T19:50:38.812 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:38.812 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:38.812 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:38.812 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:38.812 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:38.812 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:38.812 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:38.812 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:38.812 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:38.812 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:38.813 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.812+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7ff000004ee0 con 0x7ff00814be90 2024-03-25T19:50:38.813 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.813+0000 7ff0077fe640 10 monclient: get_auth_request con 0x7feff406df70 auth_method 0 2024-03-25T19:50:38.813 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.813+0000 7fefeffff640 10 client.4612.objecter ms_handle_connect 0x7feff406df70 2024-03-25T19:50:38.815 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.815+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7ff0000549e0 con 0x7ff00814be90 2024-03-25T19:50:38.815 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.815+0000 7fefeffff640 10 client.4612.objecter ms_dispatch 0x7ff008000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:38.815 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.815+0000 7fefeffff640 3 client.4612.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:38.815 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.815+0000 7fefeffff640 3 client.4612.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:38.815 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.815+0000 7fefeffff640 20 client.4612.objecter dump_active .. 0 homeless 2024-03-25T19:50:38.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.819+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff000050320 con 0x7ff00814be90 2024-03-25T19:50:38.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.819+0000 7fefeffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:38.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:38.819+0000 7fefeffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:39.048 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.048+0000 7ff00dcd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:39.049 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.048+0000 7ff00dcd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:39.049 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.048+0000 7ff00dcd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:39.049 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.048+0000 7ff00dcd6640 1 -- 172.21.15.81:0/2218125345 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff008069f40 con 0x7ff00814be90 2024-03-25T19:50:39.073 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.074+0000 7fefeffff640 1 -- 172.21.15.81:0/2218125345 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7ff000052420 con 0x7ff00814be90 2024-03-25T19:50:39.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.074+0000 7fefeffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:39.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.074+0000 7fefeffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:39.074 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:50:39.074 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.076+0000 7ff00dcd6640 10 librados: watch_flush enter 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.076+0000 7ff00dcd6640 10 librados: watch_flush exit 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.076+0000 7ff00dcd6640 20 client.4612.objecter shutdown clearing up homeless session... 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.076+0000 7ff00dcd6640 10 client.4612.objecter successfully canceled tick 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.076+0000 7ff00dcd6640 1 -- 172.21.15.81:0/2218125345 >> v1:172.21.15.98:6800/3980829775 conn(0x7feff406df70 legacy=0x7feff4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.076+0000 7ff00dcd6640 10 monclient: shutdown 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.076+0000 7ff00dcd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.076+0000 7ff00dcd6640 1 -- 172.21.15.81:0/2218125345 >> v1:172.21.15.98:6789/0 conn(0x7ff00814be90 legacy=0x7ff0081e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.077+0000 7ff00dcd6640 1 -- 172.21.15.81:0/2218125345 shutdown_connections 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.077+0000 7ff00dcd6640 1 -- 172.21.15.81:0/2218125345 >> 172.21.15.81:0/2218125345 conn(0x7ff0080617a0 msgr2=0x7ff0080b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.077+0000 7ff00dcd6640 1 -- 172.21.15.81:0/2218125345 shutdown_connections 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.077+0000 7ff00dcd6640 1 -- 172.21.15.81:0/2218125345 wait complete. 2024-03-25T19:50:39.077 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:39.077+0000 7ff00dcd6640 1 librados: shutdown 2024-03-25T19:50:39.088 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:43.424 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:09:06.066 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:43.427 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:09:06.069 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:43.477 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:09:06.119 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:43.481 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:06.116 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:43.485 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:09:06.127 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:43.490 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:06.125 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:43.491 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:09:06.132 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:43.511 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:06.146 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:43.512 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:09:06.154 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:45.894 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:46.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.009+0000 7f40f3a95640 10 monclient: get_monmap_and_config 2024-03-25T19:50:46.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.009+0000 7f40f3a95640 10 monclient: build_initial_monmap 2024-03-25T19:50:46.009 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.009+0000 7f40f3a95640 10 monclient: monmap: 2024-03-25T19:50:46.009 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:50:46.009 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:46.009 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:50:46.010345+0000 2024-03-25T19:50:46.009 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:50:46.010345+0000 2024-03-25T19:50:46.009 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:46.009 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:46.009 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:50:46.009 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:50:46.009 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:50:46.010 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:46.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.010+0000 7f40f3a95640 1 Processor -- start 2024-03-25T19:50:46.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.010+0000 7f40f3a95640 1 -- start start 2024-03-25T19:50:46.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.010+0000 7f40f3a95640 10 monclient: init 2024-03-25T19:50:46.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.010+0000 7f40f3a95640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:46.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.010+0000 7f40f3a95640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:50:46.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.010+0000 7f40f3a95640 10 monclient(hunting): picked mon.noname-a con 0x7f40ec0b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:50:46.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.011+0000 7f40f3a95640 10 monclient(hunting): picked mon.noname-c con 0x7f40ec0b1730 addr v1:172.21.15.98:6789/0 2024-03-25T19:50:46.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.011+0000 7f40f3a95640 10 monclient(hunting): picked mon.noname-b con 0x7f40ec14be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:50:46.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.011+0000 7f40f3a95640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ec0af940 con 0x7f40ec0b4d60 2024-03-25T19:50:46.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.011+0000 7f40f3a95640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ec0b0a70 con 0x7f40ec0b1730 2024-03-25T19:50:46.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.011+0000 7f40f3a95640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ec0b0d50 con 0x7f40ec14be90 2024-03-25T19:50:46.011 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.011+0000 7f40f3a95640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:46.012 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.011+0000 7f40f3a95640 10 monclient(hunting): authenticate will time out at 4481.093041s 2024-03-25T19:50:46.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.014+0000 7f40f1030640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f40ec0b1730 0x7f40ec0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:33530/0 (socket says 172.21.15.81:33530) 2024-03-25T19:50:46.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.014+0000 7f40f1030640 1 -- 172.21.15.81:0/1209717618 learned_addr learned my addr 172.21.15.81:0/1209717618 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:50:46.019 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.019+0000 7f40f082f640 1 -- 172.21.15.81:0/1209717618 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551139445 0 0) 0x7f40ec0af940 con 0x7f40ec0b4d60 2024-03-25T19:50:46.019 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.019+0000 7f40f082f640 10 monclient(hunting): _negotiate 2024-03-25T19:50:46.019 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.019+0000 7f40f082f640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:46.019 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.019+0000 7f40f082f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:46.019 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.019+0000 7f40f082f640 10 monclient(hunting): my global_id is 4629 2024-03-25T19:50:46.019 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.019+0000 7f40f082f640 1 -- 172.21.15.81:0/1209717618 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d8003520 con 0x7f40ec0b4d60 2024-03-25T19:50:46.025 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.026+0000 7f40f082f640 1 -- 172.21.15.81:0/1209717618 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811639818 0 0) 0x7f40ec0b0a70 con 0x7f40ec0b1730 2024-03-25T19:50:46.025 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.026+0000 7f40f082f640 10 monclient(hunting): _negotiate 2024-03-25T19:50:46.025 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.026+0000 7f40f082f640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:46.025 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.026+0000 7f40f082f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:46.025 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.026+0000 7f40f082f640 10 monclient(hunting): my global_id is 4615 2024-03-25T19:50:46.025 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.026+0000 7f40f082f640 1 -- 172.21.15.81:0/1209717618 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40ec0af940 con 0x7f40ec0b1730 2024-03-25T19:50:46.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.029+0000 7f40f082f640 1 -- 172.21.15.81:0/1209717618 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1561651740 0 0) 0x7f40d8003520 con 0x7f40ec0b4d60 2024-03-25T19:50:46.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.029+0000 7f40f082f640 1 -- 172.21.15.81:0/1209717618 >> v1:172.21.15.81:6790/0 conn(0x7f40ec14be90 legacy=0x7f40ec14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:50:46.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.029+0000 7f40f082f640 1 -- 172.21.15.81:0/1209717618 >> v1:172.21.15.98:6789/0 conn(0x7f40ec0b1730 legacy=0x7f40ec0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:46.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.029+0000 7f40f082f640 10 monclient: _finish_hunting 0 2024-03-25T19:50:46.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.029+0000 7f40f082f640 1 monclient: found mon.noname-a 2024-03-25T19:50:46.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.029+0000 7f40f082f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:46.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.029+0000 7f40f082f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:50:46.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.029+0000 7f40f082f640 1 -- 172.21.15.81:0/1209717618 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ec0b1030 con 0x7f40ec0b4d60 2024-03-25T19:50:46.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.029+0000 7f40f082f640 10 monclient: _finish_auth 0 2024-03-25T19:50:46.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.029+0000 7f40f082f640 10 monclient: _check_auth_tickets 2024-03-25T19:50:46.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.029+0000 7f40f082f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:46.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.029+0000 7f40f082f640 1 -- 172.21.15.81:0/1209717618 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f40e8001690 con 0x7f40ec0b4d60 2024-03-25T19:50:46.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.029+0000 7f40f082f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:46.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.029+0000 7f40f082f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:46.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.029+0000 7f40f082f640 10 monclient: dump: 2024-03-25T19:50:46.030 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:46.030 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:46.030 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:46.030 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:46.030 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:46.030 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:46.030 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:46.030 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:46.031 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:46.031 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:46.031 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.029+0000 7f40f3a95640 5 monclient: authenticate success, global_id 4629 2024-03-25T19:50:46.031 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.029+0000 7f40f3a95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:46.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.034+0000 7f40f082f640 1 -- 172.21.15.81:0/1209717618 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40e80017f0 con 0x7f40ec0b4d60 2024-03-25T19:50:46.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.034+0000 7f40f082f640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:46.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.034+0000 7f40f082f640 1 -- 172.21.15.81:0/1209717618 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f40e8001c00 con 0x7f40ec0b4d60 2024-03-25T19:50:46.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.034+0000 7f40f082f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:46.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.034+0000 7f40f082f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:46.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.034+0000 7f40f082f640 10 monclient: dump: 2024-03-25T19:50:46.035 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:46.035 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:46.035 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:46.035 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:46.035 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:46.035 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:46.035 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:46.035 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:46.035 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:46.035 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:46.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.034+0000 7f40f3a95640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:46.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.034+0000 7f40f3a95640 4 set_mon_vals no callback set 2024-03-25T19:50:46.037 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.035+0000 7f40f3a95640 10 monclient: shutdown 2024-03-25T19:50:46.038 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.035+0000 7f40f3a95640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:46.038 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.035+0000 7f40f3a95640 1 -- 172.21.15.81:0/1209717618 >> v1:172.21.15.81:6789/0 conn(0x7f40ec0b4d60 legacy=0x7f40ec13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:46.038 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.035+0000 7f40f3a95640 1 -- 172.21.15.81:0/1209717618 shutdown_connections 2024-03-25T19:50:46.038 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.035+0000 7f40f3a95640 1 -- 172.21.15.81:0/1209717618 >> 172.21.15.81:0/1209717618 conn(0x7f40ec0617a0 msgr2=0x7f40ec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:46.038 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.035+0000 7f40f3a95640 1 -- 172.21.15.81:0/1209717618 shutdown_connections 2024-03-25T19:50:46.038 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.035+0000 7f40f3a95640 1 -- 172.21.15.81:0/1209717618 wait complete. 2024-03-25T19:50:46.038 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.036+0000 7f40f3a95640 10 monclient: build_initial_monmap 2024-03-25T19:50:46.038 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.036+0000 7f40f3a95640 10 monclient: monmap: 2024-03-25T19:50:46.038 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:50:46.038 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:46.038 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:50:46.039 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:50:46.039 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:46.039 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:50:46.039 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:50:46.039 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:50:46.039 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:50:46.039 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:46.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.036+0000 7f40f3a95640 1 librados: starting msgr at 2024-03-25T19:50:46.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.036+0000 7f40f3a95640 1 librados: starting objecter 2024-03-25T19:50:46.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.036+0000 7f40f3a95640 1 Processor -- start 2024-03-25T19:50:46.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.037+0000 7f40f3a95640 1 -- start start 2024-03-25T19:50:46.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.037+0000 7f40f3a95640 1 librados: setting wanted keys 2024-03-25T19:50:46.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.037+0000 7f40f3a95640 1 librados: calling monclient init 2024-03-25T19:50:46.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.037+0000 7f40f3a95640 10 monclient: init 2024-03-25T19:50:46.040 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.037+0000 7f40f3a95640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:46.040 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.037+0000 7f40f3a95640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:50:46.040 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.037+0000 7f40f3a95640 10 monclient(hunting): picked mon.noname-a con 0x7f40ec0b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:50:46.040 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.037+0000 7f40f3a95640 10 monclient(hunting): picked mon.noname-b con 0x7f40ec0b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:50:46.040 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.037+0000 7f40f3a95640 10 monclient(hunting): picked mon.noname-c con 0x7f40ec14be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:50:46.040 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.037+0000 7f40f3a95640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ec138840 con 0x7f40ec0b1730 2024-03-25T19:50:46.040 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.037+0000 7f40f3a95640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ec1d7960 con 0x7f40ec0b4d60 2024-03-25T19:50:46.040 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.037+0000 7f40f3a95640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ec138e60 con 0x7f40ec14be90 2024-03-25T19:50:46.040 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.037+0000 7f40f3a95640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:46.040 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.037+0000 7f40f3a95640 10 monclient(hunting): authenticate will time out at 4481.119603s 2024-03-25T19:50:46.045 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.045+0000 7f40f1030640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f40ec0b4d60 0x7f40ec137a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:33564/0 (socket says 172.21.15.81:33564) 2024-03-25T19:50:46.045 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.045+0000 7f40f1030640 1 -- 172.21.15.81:0/2008806921 learned_addr learned my addr 172.21.15.81:0/2008806921 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:50:46.048 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.048+0000 7f40e27fc640 10 client.?.objecter ms_handle_connect 0x7f40ec0b4d60 2024-03-25T19:50:46.048 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.048+0000 7f40e27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:46.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.056+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40192678 0 0) 0x7f40ec1d7960 con 0x7f40ec0b4d60 2024-03-25T19:50:46.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.056+0000 7f40e27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:46.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.056+0000 7f40e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:46.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.056+0000 7f40e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:46.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.056+0000 7f40e27fc640 10 monclient(hunting): my global_id is 4618 2024-03-25T19:50:46.056 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.056+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d0004eb0 con 0x7f40ec0b4d60 2024-03-25T19:50:46.057 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.058+0000 7f40e27fc640 10 client.?.objecter ms_handle_connect 0x7f40ec0b1730 2024-03-25T19:50:46.057 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.058+0000 7f40e27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:46.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.060+0000 7f40e27fc640 10 client.?.objecter ms_handle_connect 0x7f40ec14be90 2024-03-25T19:50:46.060 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.060+0000 7f40e27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:46.070 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.070+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4151090946 0 0) 0x7f40d0004eb0 con 0x7f40ec0b4d60 2024-03-25T19:50:46.070 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.070+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40ec1d7960 con 0x7f40ec0b4d60 2024-03-25T19:50:46.070 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.070+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f40dc0019f0 con 0x7f40ec0b4d60 2024-03-25T19:50:46.070 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.070+0000 7f40e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:46.070 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.070+0000 7f40e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:46.070 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.070+0000 7f40e27fc640 10 monclient(hunting): dump: 2024-03-25T19:50:46.070 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:46.070 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:46.070 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:46.070 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:46.071 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:46.071 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:46.071 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:46.071 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:46.071 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:46.071 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:46.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.070+0000 7f40e27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:46.071 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.071+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859283830 0 0) 0x7f40ec138e60 con 0x7f40ec14be90 2024-03-25T19:50:46.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.071+0000 7f40e27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:46.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.071+0000 7f40e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:46.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.071+0000 7f40e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:46.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.071+0000 7f40e27fc640 10 monclient(hunting): my global_id is 4598 2024-03-25T19:50:46.072 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.071+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d0004eb0 con 0x7f40ec14be90 2024-03-25T19:50:46.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.076+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809061321 0 0) 0x7f40ec138840 con 0x7f40ec0b1730 2024-03-25T19:50:46.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.076+0000 7f40e27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:46.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.076+0000 7f40e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:46.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.076+0000 7f40e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:46.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.076+0000 7f40e27fc640 10 monclient(hunting): my global_id is 4632 2024-03-25T19:50:46.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.076+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40ec138e60 con 0x7f40ec0b1730 2024-03-25T19:50:46.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2913860554 0 0) 0x7f40ec1d7960 con 0x7f40ec0b4d60 2024-03-25T19:50:46.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 >> v1:172.21.15.81:6790/0 conn(0x7f40ec14be90 legacy=0x7f40ec138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:46.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 >> v1:172.21.15.81:6789/0 conn(0x7f40ec0b1730 legacy=0x7f40ec1d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:46.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40e27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:46.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40e27fc640 1 monclient: found mon.b 2024-03-25T19:50:46.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:46.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40e27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:46.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ec138ff0 con 0x7f40ec0b4d60 2024-03-25T19:50:46.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40e27fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:46.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40e27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:46.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:46.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40f3a95640 5 monclient: authenticate success, global_id 4618 2024-03-25T19:50:46.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40f3a95640 10 monclient: _renew_subs 2024-03-25T19:50:46.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40f3a95640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:46.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40f3a95640 1 -- 172.21.15.81:0/2008806921 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40ec1d8550 con 0x7f40ec0b4d60 2024-03-25T19:50:46.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40f3a95640 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:46.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40f3a95640 10 monclient: _renew_subs 2024-03-25T19:50:46.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40f3a95640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:46.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40f3a95640 1 -- 172.21.15.81:0/2008806921 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ec13bbe0 con 0x7f40ec0b4d60 2024-03-25T19:50:46.083 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.081+0000 7f40f3a95640 1 librados: init done 2024-03-25T19:50:46.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.082+0000 7f40f3a95640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:46.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.082+0000 7f40f3a95640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:46.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.082+0000 7f40f3a95640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:46.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.082+0000 7f40f3a95640 1 -- 172.21.15.81:0/2008806921 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ec0627b0 con 0x7f40ec0b4d60 2024-03-25T19:50:46.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.350+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40dc001c70 con 0x7f40ec0b4d60 2024-03-25T19:50:46.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.350+0000 7f40e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:46.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.350+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f40dc004c30 con 0x7f40ec0b4d60 2024-03-25T19:50:46.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.350+0000 7f40e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:46.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.350+0000 7f40e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:46.351 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.350+0000 7f40e27fc640 10 monclient: dump: 2024-03-25T19:50:46.351 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:46.351 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:46.351 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:46.351 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:46.351 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:46.352 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:46.352 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:46.352 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:46.352 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:46.352 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:46.352 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.350+0000 7f40e3fff640 4 set_mon_vals no callback set 2024-03-25T19:50:46.352 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.351+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f40dc01ac00 con 0x7f40ec0b4d60 2024-03-25T19:50:46.352 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.352+0000 7f40f1831640 10 monclient: get_auth_request con 0x7f40d006de80 auth_method 0 2024-03-25T19:50:46.352 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.352+0000 7f40e27fc640 10 client.4618.objecter ms_handle_connect 0x7f40d006de80 2024-03-25T19:50:46.358 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.359+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f40dc0548c0 con 0x7f40ec0b4d60 2024-03-25T19:50:46.359 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.359+0000 7f40e27fc640 10 client.4618.objecter ms_dispatch 0x7f40ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:46.359 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.359+0000 7f40e27fc640 3 client.4618.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:46.359 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.359+0000 7f40e27fc640 3 client.4618.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:46.359 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.359+0000 7f40e27fc640 20 client.4618.objecter dump_active .. 0 homeless 2024-03-25T19:50:46.359 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.360+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f40dc050200 con 0x7f40ec0b4d60 2024-03-25T19:50:46.359 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.360+0000 7f40e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:46.359 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.360+0000 7f40e27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:46.617 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.617+0000 7f40f3a95640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:46.617 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.617+0000 7f40f3a95640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:46.617 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.617+0000 7f40f3a95640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:50:46.617 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.617+0000 7f40f3a95640 1 -- 172.21.15.81:0/2008806921 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40ec069f40 con 0x7f40ec0b4d60 2024-03-25T19:50:46.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.635+0000 7f40e27fc640 1 -- 172.21.15.81:0/2008806921 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f40dc052300 con 0x7f40ec0b4d60 2024-03-25T19:50:46.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.635+0000 7f40e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:46.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.635+0000 7f40e27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:46.635 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:50:46.636 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.637+0000 7f40f3a95640 10 librados: watch_flush enter 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.637+0000 7f40f3a95640 10 librados: watch_flush exit 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.637+0000 7f40f3a95640 20 client.4618.objecter shutdown clearing up homeless session... 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.637+0000 7f40f3a95640 10 client.4618.objecter successfully canceled tick 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.638+0000 7f40f3a95640 1 -- 172.21.15.81:0/2008806921 >> v1:172.21.15.98:6800/3980829775 conn(0x7f40d006de80 legacy=0x7f40d0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.638+0000 7f40f3a95640 10 monclient: shutdown 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.638+0000 7f40f3a95640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.638+0000 7f40f3a95640 1 -- 172.21.15.81:0/2008806921 >> v1:172.21.15.98:6789/0 conn(0x7f40ec0b4d60 legacy=0x7f40ec137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.638+0000 7f40f3a95640 1 -- 172.21.15.81:0/2008806921 shutdown_connections 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.638+0000 7f40f3a95640 1 -- 172.21.15.81:0/2008806921 >> 172.21.15.81:0/2008806921 conn(0x7f40ec0617a0 msgr2=0x7f40ec145020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.638+0000 7f40f3a95640 1 -- 172.21.15.81:0/2008806921 shutdown_connections 2024-03-25T19:50:46.638 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.638+0000 7f40f3a95640 1 -- 172.21.15.81:0/2008806921 wait complete. 2024-03-25T19:50:46.639 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:46.638+0000 7f40f3a95640 1 librados: shutdown 2024-03-25T19:50:46.650 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:53.457 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:53.564 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.564+0000 7fdc43df6640 10 monclient: get_monmap_and_config 2024-03-25T19:50:53.564 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.564+0000 7fdc43df6640 10 monclient: build_initial_monmap 2024-03-25T19:50:53.564 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.564+0000 7fdc43df6640 10 monclient: monmap: 2024-03-25T19:50:53.564 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:50:53.565 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:53.565 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:50:53.565817+0000 2024-03-25T19:50:53.565 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:50:53.565817+0000 2024-03-25T19:50:53.565 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:53.565 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:53.565 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:50:53.565 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:50:53.565 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:50:53.565 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:53.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.565+0000 7fdc43df6640 1 Processor -- start 2024-03-25T19:50:53.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.565+0000 7fdc43df6640 1 -- start start 2024-03-25T19:50:53.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.565+0000 7fdc43df6640 10 monclient: init 2024-03-25T19:50:53.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.566+0000 7fdc43df6640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:53.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.566+0000 7fdc43df6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:53.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.566+0000 7fdc43df6640 10 monclient(hunting): picked mon.noname-a con 0x7fdc3c0b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:50:53.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.566+0000 7fdc43df6640 10 monclient(hunting): picked mon.noname-b con 0x7fdc3c0b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:50:53.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.566+0000 7fdc43df6640 10 monclient(hunting): picked mon.noname-c con 0x7fdc3c14be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:50:53.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.566+0000 7fdc43df6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc3c0af940 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.566+0000 7fdc43df6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc3c0b0a70 con 0x7fdc3c14be90 2024-03-25T19:50:53.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.566+0000 7fdc43df6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc3c0b0d50 con 0x7fdc3c0b1730 2024-03-25T19:50:53.567 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.566+0000 7fdc43df6640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:53.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.566+0000 7fdc43df6640 10 monclient(hunting): authenticate will time out at 4488.648103s 2024-03-25T19:50:53.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.566+0000 7fdc41391640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fdc3c0b1730 0x7fdc3c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:37972/0 (socket says 172.21.15.81:37972) 2024-03-25T19:50:53.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.566+0000 7fdc41391640 1 -- 172.21.15.81:0/1925605931 learned_addr learned my addr 172.21.15.81:0/1925605931 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:50:53.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.625+0000 7fdc40b90640 1 -- 172.21.15.81:0/1925605931 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952706510 0 0) 0x7fdc3c0af940 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.625+0000 7fdc40b90640 10 monclient(hunting): _negotiate 2024-03-25T19:50:53.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.625+0000 7fdc40b90640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:53.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.625+0000 7fdc40b90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:53.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.625+0000 7fdc40b90640 10 monclient(hunting): my global_id is 4635 2024-03-25T19:50:53.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.625+0000 7fdc40b90640 1 -- 172.21.15.81:0/1925605931 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc28003520 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc40b90640 1 -- 172.21.15.81:0/1925605931 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1698816342 0 0) 0x7fdc28003520 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc40b90640 1 -- 172.21.15.81:0/1925605931 >> v1:172.21.15.81:6790/0 conn(0x7fdc3c0b1730 legacy=0x7fdc3c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:53.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc40b90640 1 -- 172.21.15.81:0/1925605931 >> v1:172.21.15.98:6789/0 conn(0x7fdc3c14be90 legacy=0x7fdc3c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:53.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc40b90640 10 monclient: _finish_hunting 0 2024-03-25T19:50:53.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc40b90640 1 monclient: found mon.noname-a 2024-03-25T19:50:53.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc40b90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:53.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc40b90640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:50:53.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc40b90640 1 -- 172.21.15.81:0/1925605931 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3c0b1030 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc40b90640 10 monclient: _finish_auth 0 2024-03-25T19:50:53.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc40b90640 10 monclient: _check_auth_tickets 2024-03-25T19:50:53.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc40b90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:53.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc40b90640 1 -- 172.21.15.81:0/1925605931 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fdc34001690 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc43df6640 5 monclient: authenticate success, global_id 4635 2024-03-25T19:50:53.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc43df6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:53.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc40b90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:53.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc40b90640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:53.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc40b90640 10 monclient: dump: 2024-03-25T19:50:53.647 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:53.647 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:53.647 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:53.647 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:53.647 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:53.647 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:53.647 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:53.647 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:53.647 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:53.648 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:53.648 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.646+0000 7fdc43df6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:53.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.656+0000 7fdc40b90640 1 -- 172.21.15.81:0/1925605931 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc340017f0 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.656+0000 7fdc40b90640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:53.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.656+0000 7fdc40b90640 1 -- 172.21.15.81:0/1925605931 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fdc34001c00 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.656+0000 7fdc40b90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:53.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.656+0000 7fdc40b90640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:53.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.656+0000 7fdc40b90640 10 monclient: dump: 2024-03-25T19:50:53.656 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:53.656 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:53.656 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:53.656 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:53.656 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:53.657 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:53.657 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:53.657 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:53.657 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:53.657 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:53.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.656+0000 7fdc43df6640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:53.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.656+0000 7fdc43df6640 4 set_mon_vals no callback set 2024-03-25T19:50:53.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.657+0000 7fdc43df6640 10 monclient: shutdown 2024-03-25T19:50:53.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.657+0000 7fdc43df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:53.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.657+0000 7fdc43df6640 1 -- 172.21.15.81:0/1925605931 >> v1:172.21.15.81:6789/0 conn(0x7fdc3c0b4d60 legacy=0x7fdc3c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:53.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.657+0000 7fdc43df6640 1 -- 172.21.15.81:0/1925605931 shutdown_connections 2024-03-25T19:50:53.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.657+0000 7fdc43df6640 1 -- 172.21.15.81:0/1925605931 >> 172.21.15.81:0/1925605931 conn(0x7fdc3c0617a0 msgr2=0x7fdc3c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:53.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.657+0000 7fdc43df6640 1 -- 172.21.15.81:0/1925605931 shutdown_connections 2024-03-25T19:50:53.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.657+0000 7fdc43df6640 1 -- 172.21.15.81:0/1925605931 wait complete. 2024-03-25T19:50:53.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.658+0000 7fdc43df6640 10 monclient: build_initial_monmap 2024-03-25T19:50:53.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.658+0000 7fdc43df6640 10 monclient: monmap: 2024-03-25T19:50:53.660 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:50:53.660 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:53.660 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:50:53.660 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:50:53.660 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:53.660 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:50:53.660 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:50:53.660 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:50:53.660 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:50:53.660 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:53.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.658+0000 7fdc43df6640 1 librados: starting msgr at 2024-03-25T19:50:53.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.658+0000 7fdc43df6640 1 librados: starting objecter 2024-03-25T19:50:53.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.658+0000 7fdc43df6640 1 Processor -- start 2024-03-25T19:50:53.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.658+0000 7fdc43df6640 1 -- start start 2024-03-25T19:50:53.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.658+0000 7fdc43df6640 1 librados: setting wanted keys 2024-03-25T19:50:53.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.658+0000 7fdc43df6640 1 librados: calling monclient init 2024-03-25T19:50:53.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.658+0000 7fdc43df6640 10 monclient: init 2024-03-25T19:50:53.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.659+0000 7fdc43df6640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:53.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.659+0000 7fdc43df6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:53.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.659+0000 7fdc43df6640 10 monclient(hunting): picked mon.noname-b con 0x7fdc3c0b1730 addr v1:172.21.15.98:6789/0 2024-03-25T19:50:53.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.659+0000 7fdc43df6640 10 monclient(hunting): picked mon.noname-c con 0x7fdc3c0b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:50:53.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.659+0000 7fdc43df6640 10 monclient(hunting): picked mon.noname-a con 0x7fdc3c14be90 addr v1:172.21.15.81:6789/0 2024-03-25T19:50:53.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.659+0000 7fdc43df6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc3c13a940 con 0x7fdc3c14be90 2024-03-25T19:50:53.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.659+0000 7fdc43df6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc3c13ab90 con 0x7fdc3c0b1730 2024-03-25T19:50:53.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.659+0000 7fdc43df6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc3c1d7840 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.659+0000 7fdc43df6640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:53.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.659+0000 7fdc43df6640 10 monclient(hunting): authenticate will time out at 4488.741221s 2024-03-25T19:50:53.665 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.666+0000 7fdc41391640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fdc3c0b4d60 0x7fdc3c1d7d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:38000/0 (socket says 172.21.15.81:38000) 2024-03-25T19:50:53.666 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.666+0000 7fdc41391640 1 -- 172.21.15.81:0/3012299540 learned_addr learned my addr 172.21.15.81:0/3012299540 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:50:53.667 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.667+0000 7fdc327fc640 10 client.?.objecter ms_handle_connect 0x7fdc3c0b4d60 2024-03-25T19:50:53.667 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.667+0000 7fdc327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:53.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.673+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840247734 0 0) 0x7fdc3c1d7840 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.673+0000 7fdc327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:53.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.673+0000 7fdc327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:53.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.673+0000 7fdc327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:53.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.673+0000 7fdc327fc640 10 monclient(hunting): my global_id is 4604 2024-03-25T19:50:53.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.673+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc20004eb0 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.679 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.679+0000 7fdc327fc640 10 client.?.objecter ms_handle_connect 0x7fdc3c0b1730 2024-03-25T19:50:53.679 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.679+0000 7fdc327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:53.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.681+0000 7fdc327fc640 10 client.?.objecter ms_handle_connect 0x7fdc3c14be90 2024-03-25T19:50:53.681 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.681+0000 7fdc327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:53.681 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.681+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1046641014 0 0) 0x7fdc20004eb0 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.681 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.681+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc3c1d7840 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.682 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.683+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fdc2c001990 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.682 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.683+0000 7fdc327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:53.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.683+0000 7fdc327fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:53.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.683+0000 7fdc327fc640 10 monclient(hunting): dump: 2024-03-25T19:50:53.683 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:53.683 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:53.683 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:53.683 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:53.683 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:53.683 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:53.683 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:53.683 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:53.683 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:53.683 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:53.683 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.683+0000 7fdc327fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:53.688 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.689+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131691774 0 0) 0x7fdc3c13a940 con 0x7fdc3c14be90 2024-03-25T19:50:53.689 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.689+0000 7fdc327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:53.689 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.689+0000 7fdc327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:53.689 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.689+0000 7fdc327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:53.689 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.689+0000 7fdc327fc640 10 monclient(hunting): my global_id is 4638 2024-03-25T19:50:53.689 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.689+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc20004eb0 con 0x7fdc3c14be90 2024-03-25T19:50:53.692 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.692+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319414724 0 0) 0x7fdc3c13ab90 con 0x7fdc3c0b1730 2024-03-25T19:50:53.692 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.692+0000 7fdc327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:53.692 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.692+0000 7fdc327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:53.692 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.692+0000 7fdc327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:53.692 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.692+0000 7fdc327fc640 10 monclient(hunting): my global_id is 4624 2024-03-25T19:50:53.692 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.692+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc3c13a940 con 0x7fdc3c0b1730 2024-03-25T19:50:53.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4141405336 0 0) 0x7fdc3c1d7840 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 >> v1:172.21.15.98:6789/0 conn(0x7fdc3c0b1730 legacy=0x7fdc3c13bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:53.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 >> v1:172.21.15.81:6789/0 conn(0x7fdc3c14be90 legacy=0x7fdc3c1e3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:53.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc327fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:53.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc327fc640 1 monclient: found mon.c 2024-03-25T19:50:53.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:53.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc327fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:50:53.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3c13c240 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc327fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:53.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc327fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:53.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:53.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc43df6640 5 monclient: authenticate success, global_id 4604 2024-03-25T19:50:53.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc43df6640 10 monclient: _renew_subs 2024-03-25T19:50:53.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc43df6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:50:53.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc43df6640 1 -- 172.21.15.81:0/3012299540 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc3c139ce0 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc43df6640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:53.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc43df6640 10 monclient: _renew_subs 2024-03-25T19:50:53.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc43df6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:50:53.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc43df6640 1 -- 172.21.15.81:0/3012299540 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc3c1d8650 con 0x7fdc3c0b4d60 2024-03-25T19:50:53.705 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.704+0000 7fdc43df6640 1 librados: init done 2024-03-25T19:50:53.706 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.705+0000 7fdc43df6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:53.706 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.705+0000 7fdc43df6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:53.706 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.705+0000 7fdc43df6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:50:53.706 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:53.705+0000 7fdc43df6640 1 -- 172.21.15.81:0/3012299540 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc3c1401f0 con 0x7fdc3c0b4d60 2024-03-25T19:50:54.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.016+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc2c0047e0 con 0x7fdc3c0b4d60 2024-03-25T19:50:54.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.016+0000 7fdc327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:54.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.016+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fdc2c004bb0 con 0x7fdc3c0b4d60 2024-03-25T19:50:54.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.016+0000 7fdc327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:54.016 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.016+0000 7fdc327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:54.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.016+0000 7fdc327fc640 10 monclient: dump: 2024-03-25T19:50:54.017 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:50:54.017 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:50:54.017 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:54.017 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:50:54.017 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:54.017 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:50:54.017 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:50:54.017 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:50:54.017 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:50:54.017 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:50:54.017 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.016+0000 7fdc33fff640 4 set_mon_vals no callback set 2024-03-25T19:50:54.018 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.017+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fdc2c004dd0 con 0x7fdc3c0b4d60 2024-03-25T19:50:54.018 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.017+0000 7fdc41b92640 10 monclient: get_auth_request con 0x7fdc2006dc40 auth_method 0 2024-03-25T19:50:54.018 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.018+0000 7fdc327fc640 10 client.4604.objecter ms_handle_connect 0x7fdc2006dc40 2024-03-25T19:50:54.023 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.023+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fdc2c0548a0 con 0x7fdc3c0b4d60 2024-03-25T19:50:54.023 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.023+0000 7fdc327fc640 10 client.4604.objecter ms_dispatch 0x7fdc3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:54.023 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.023+0000 7fdc327fc640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:54.023 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.023+0000 7fdc327fc640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:54.023 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.023+0000 7fdc327fc640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-25T19:50:54.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.023+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc2c0503b0 con 0x7fdc3c0b4d60 2024-03-25T19:50:54.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.024+0000 7fdc327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:54.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.024+0000 7fdc327fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:54.268 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.269+0000 7fdc43df6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:54.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.269+0000 7fdc43df6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:54.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.269+0000 7fdc43df6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:50:54.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.269+0000 7fdc43df6640 1 -- 172.21.15.81:0/3012299540 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc3c0627b0 con 0x7fdc3c0b4d60 2024-03-25T19:50:54.297 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.297+0000 7fdc327fc640 1 -- 172.21.15.81:0/3012299540 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fdc2c0522e0 con 0x7fdc3c0b4d60 2024-03-25T19:50:54.297 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.297+0000 7fdc327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:54.297 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.297+0000 7fdc327fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:54.297 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:50:54.297 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:54.300 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.300+0000 7fdc43df6640 10 librados: watch_flush enter 2024-03-25T19:50:54.300 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.300+0000 7fdc43df6640 10 librados: watch_flush exit 2024-03-25T19:50:54.301 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.300+0000 7fdc43df6640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-25T19:50:54.301 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.300+0000 7fdc43df6640 10 client.4604.objecter successfully canceled tick 2024-03-25T19:50:54.301 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.300+0000 7fdc43df6640 1 -- 172.21.15.81:0/3012299540 >> v1:172.21.15.98:6800/3980829775 conn(0x7fdc2006dc40 legacy=0x7fdc20070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:54.301 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.300+0000 7fdc43df6640 10 monclient: shutdown 2024-03-25T19:50:54.301 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.300+0000 7fdc43df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:54.301 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.300+0000 7fdc43df6640 1 -- 172.21.15.81:0/3012299540 >> v1:172.21.15.81:6790/0 conn(0x7fdc3c0b4d60 legacy=0x7fdc3c1d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:54.301 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.300+0000 7fdc43df6640 1 -- 172.21.15.81:0/3012299540 shutdown_connections 2024-03-25T19:50:54.301 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.300+0000 7fdc43df6640 1 -- 172.21.15.81:0/3012299540 >> 172.21.15.81:0/3012299540 conn(0x7fdc3c0617a0 msgr2=0x7fdc3c137510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:54.301 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.300+0000 7fdc43df6640 1 -- 172.21.15.81:0/3012299540 shutdown_connections 2024-03-25T19:50:54.301 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.300+0000 7fdc43df6640 1 -- 172.21.15.81:0/3012299540 wait complete. 2024-03-25T19:50:54.301 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:50:54.300+0000 7fdc43df6640 1 librados: shutdown 2024-03-25T19:50:54.311 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:01.119 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:01.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.237+0000 7f53bc63e640 10 monclient: get_monmap_and_config 2024-03-25T19:51:01.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.237+0000 7f53bc63e640 10 monclient: build_initial_monmap 2024-03-25T19:51:01.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.237+0000 7f53bc63e640 10 monclient: monmap: 2024-03-25T19:51:01.237 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:51:01.237 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:01.237 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:51:01.237942+0000 2024-03-25T19:51:01.237 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:51:01.237942+0000 2024-03-25T19:51:01.237 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:01.237 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:01.238 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:51:01.238 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:51:01.238 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:51:01.238 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:01.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.237+0000 7f53bc63e640 1 Processor -- start 2024-03-25T19:51:01.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.237+0000 7f53bc63e640 1 -- start start 2024-03-25T19:51:01.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.237+0000 7f53bc63e640 10 monclient: init 2024-03-25T19:51:01.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.238+0000 7f53bc63e640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:01.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.238+0000 7f53bc63e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:51:01.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.238+0000 7f53bc63e640 10 monclient(hunting): picked mon.noname-b con 0x7f53b40a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:51:01.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.238+0000 7f53bc63e640 10 monclient(hunting): picked mon.noname-c con 0x7f53b413f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:51:01.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.238+0000 7f53bc63e640 10 monclient(hunting): picked mon.noname-a con 0x7f53b41430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:51:01.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.238+0000 7f53bc63e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b4142520 con 0x7f53b41430c0 2024-03-25T19:51:01.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.238+0000 7f53bc63e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b40b28a0 con 0x7f53b413f710 2024-03-25T19:51:01.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.238+0000 7f53bc63e640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b40b2a30 con 0x7f53b40a98f0 2024-03-25T19:51:01.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.238+0000 7f53bc63e640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:01.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.238+0000 7f53bc63e640 10 monclient(hunting): authenticate will time out at 4496.320563s 2024-03-25T19:51:01.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.240+0000 7f53babdb640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f53b41430c0 0x7f53b414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:51030/0 (socket says 172.21.15.81:51030) 2024-03-25T19:51:01.240 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.240+0000 7f53babdb640 1 -- 172.21.15.81:0/2783580289 learned_addr learned my addr 172.21.15.81:0/2783580289 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:51:01.248 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.249+0000 7f53b93d8640 1 -- 172.21.15.81:0/2783580289 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2370059340 0 0) 0x7f53b4142520 con 0x7f53b41430c0 2024-03-25T19:51:01.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.249+0000 7f53b93d8640 10 monclient(hunting): _negotiate 2024-03-25T19:51:01.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.249+0000 7f53b93d8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:01.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.249+0000 7f53b93d8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:01.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.249+0000 7f53b93d8640 10 monclient(hunting): my global_id is 4641 2024-03-25T19:51:01.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.249+0000 7f53b93d8640 1 -- 172.21.15.81:0/2783580289 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a0003520 con 0x7f53b41430c0 2024-03-25T19:51:01.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53b93d8640 1 -- 172.21.15.81:0/2783580289 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 701233352 0 0) 0x7f53a0003520 con 0x7f53b41430c0 2024-03-25T19:51:01.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53b93d8640 1 -- 172.21.15.81:0/2783580289 >> v1:172.21.15.81:6790/0 conn(0x7f53b40a98f0 legacy=0x7f53b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:01.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53b93d8640 1 -- 172.21.15.81:0/2783580289 >> v1:172.21.15.98:6789/0 conn(0x7f53b413f710 legacy=0x7f53b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:01.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53b93d8640 10 monclient: _finish_hunting 0 2024-03-25T19:51:01.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53b93d8640 1 monclient: found mon.noname-a 2024-03-25T19:51:01.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53b93d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:01.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53b93d8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:51:01.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53b93d8640 1 -- 172.21.15.81:0/2783580289 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b40b44c0 con 0x7f53b41430c0 2024-03-25T19:51:01.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53b93d8640 10 monclient: _finish_auth 0 2024-03-25T19:51:01.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53b93d8640 10 monclient: _check_auth_tickets 2024-03-25T19:51:01.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53b93d8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:01.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53b93d8640 1 -- 172.21.15.81:0/2783580289 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f53ac001a10 con 0x7f53b41430c0 2024-03-25T19:51:01.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53bc63e640 5 monclient: authenticate success, global_id 4641 2024-03-25T19:51:01.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53bc63e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:01.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53b93d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:01.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53b93d8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:01.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53b93d8640 10 monclient: dump: 2024-03-25T19:51:01.259 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:01.259 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:01.259 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:01.260 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:01.260 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:01.260 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:01.260 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:01.260 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:01.260 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:01.260 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:01.260 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.258+0000 7f53bc63e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:01.266 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.266+0000 7f53b93d8640 1 -- 172.21.15.81:0/2783580289 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53ac001b70 con 0x7f53b41430c0 2024-03-25T19:51:01.266 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.266+0000 7f53b93d8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:01.266 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.266+0000 7f53b93d8640 1 -- 172.21.15.81:0/2783580289 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f53ac001f80 con 0x7f53b41430c0 2024-03-25T19:51:01.266 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.266+0000 7f53bc63e640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:01.266 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.266+0000 7f53bc63e640 4 set_mon_vals no callback set 2024-03-25T19:51:01.266 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.266+0000 7f53b93d8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:01.267 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.266+0000 7f53b93d8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:01.267 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.266+0000 7f53b93d8640 10 monclient: dump: 2024-03-25T19:51:01.267 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:01.267 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:01.267 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:01.267 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:01.267 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:01.267 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:01.267 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:01.267 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:01.267 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:01.267 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:01.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.267+0000 7f53bc63e640 10 monclient: shutdown 2024-03-25T19:51:01.269 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.267+0000 7f53bc63e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:01.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.267+0000 7f53bc63e640 1 -- 172.21.15.81:0/2783580289 >> v1:172.21.15.81:6789/0 conn(0x7f53b41430c0 legacy=0x7f53b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:01.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.267+0000 7f53bc63e640 1 -- 172.21.15.81:0/2783580289 shutdown_connections 2024-03-25T19:51:01.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.267+0000 7f53bc63e640 1 -- 172.21.15.81:0/2783580289 >> 172.21.15.81:0/2783580289 conn(0x7f53b40b7ea0 msgr2=0x7f53b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:01.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.267+0000 7f53bc63e640 1 -- 172.21.15.81:0/2783580289 shutdown_connections 2024-03-25T19:51:01.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.267+0000 7f53bc63e640 1 -- 172.21.15.81:0/2783580289 wait complete. 2024-03-25T19:51:01.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.268+0000 7f53bc63e640 10 monclient: build_initial_monmap 2024-03-25T19:51:01.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.268+0000 7f53bc63e640 10 monclient: monmap: 2024-03-25T19:51:01.270 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:51:01.270 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:01.270 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:51:01.270 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:51:01.270 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:01.270 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:51:01.271 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:51:01.271 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:51:01.271 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:51:01.271 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:01.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.268+0000 7f53bc63e640 1 librados: starting msgr at 2024-03-25T19:51:01.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.268+0000 7f53bc63e640 1 librados: starting objecter 2024-03-25T19:51:01.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.268+0000 7f53bc63e640 1 Processor -- start 2024-03-25T19:51:01.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.268+0000 7f53bc63e640 1 -- start start 2024-03-25T19:51:01.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.268+0000 7f53bc63e640 1 librados: setting wanted keys 2024-03-25T19:51:01.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.268+0000 7f53bc63e640 1 librados: calling monclient init 2024-03-25T19:51:01.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.268+0000 7f53bc63e640 10 monclient: init 2024-03-25T19:51:01.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.269+0000 7f53bc63e640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:01.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.269+0000 7f53bc63e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:01.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.269+0000 7f53bc63e640 10 monclient(hunting): picked mon.noname-a con 0x7f53b40a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:51:01.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.269+0000 7f53bc63e640 10 monclient(hunting): picked mon.noname-c con 0x7f53b413f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:51:01.272 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.269+0000 7f53bc63e640 10 monclient(hunting): picked mon.noname-b con 0x7f53b41430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:51:01.272 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.269+0000 7f53bc63e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b4137f10 con 0x7f53b40a98f0 2024-03-25T19:51:01.272 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.269+0000 7f53bc63e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b41dc070 con 0x7f53b41430c0 2024-03-25T19:51:01.272 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.269+0000 7f53bc63e640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b4137100 con 0x7f53b413f710 2024-03-25T19:51:01.272 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.269+0000 7f53bc63e640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:01.272 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.269+0000 7f53bc63e640 10 monclient(hunting): authenticate will time out at 4496.351414s 2024-03-25T19:51:01.272 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.272+0000 7f53babdb640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f53b41430c0 0x7f53b41db960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:38524/0 (socket says 172.21.15.81:38524) 2024-03-25T19:51:01.272 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.272+0000 7f53babdb640 1 -- 172.21.15.81:0/1490330142 learned_addr learned my addr 172.21.15.81:0/1490330142 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:51:01.275 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.276+0000 7f53aaffd640 10 client.?.objecter ms_handle_connect 0x7f53b40a98f0 2024-03-25T19:51:01.275 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.276+0000 7f53aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:01.276 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.276+0000 7f53aaffd640 10 client.?.objecter ms_handle_connect 0x7f53b41430c0 2024-03-25T19:51:01.276 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.276+0000 7f53aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:01.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.280+0000 7f53aaffd640 10 client.?.objecter ms_handle_connect 0x7f53b413f710 2024-03-25T19:51:01.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.280+0000 7f53aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:01.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.281+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340213514 0 0) 0x7f53b4137f10 con 0x7f53b40a98f0 2024-03-25T19:51:01.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.281+0000 7f53aaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:01.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.281+0000 7f53aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:01.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.281+0000 7f53aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:01.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.281+0000 7f53aaffd640 10 monclient(hunting): my global_id is 4644 2024-03-25T19:51:01.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.281+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5398004eb0 con 0x7f53b40a98f0 2024-03-25T19:51:01.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.293+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263523609 0 0) 0x7f53b4137100 con 0x7f53b413f710 2024-03-25T19:51:01.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.293+0000 7f53aaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:01.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.293+0000 7f53aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:01.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.293+0000 7f53aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:01.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.293+0000 7f53aaffd640 10 monclient(hunting): my global_id is 4610 2024-03-25T19:51:01.293 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.293+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b4137f10 con 0x7f53b413f710 2024-03-25T19:51:01.296 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.296+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625985689 0 0) 0x7f53b41dc070 con 0x7f53b41430c0 2024-03-25T19:51:01.296 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.296+0000 7f53aaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:01.296 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.296+0000 7f53aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:01.296 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.296+0000 7f53aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:01.296 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.296+0000 7f53aaffd640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:51:01.297 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.296+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b4137100 con 0x7f53b41430c0 2024-03-25T19:51:01.307 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.307+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1169849239 0 0) 0x7f5398004eb0 con 0x7f53b40a98f0 2024-03-25T19:51:01.307 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.307+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53b41dc070 con 0x7f53b40a98f0 2024-03-25T19:51:01.307 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.307+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f53b0001930 con 0x7f53b40a98f0 2024-03-25T19:51:01.307 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.307+0000 7f53aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:01.307 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.307+0000 7f53aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:01.307 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.307+0000 7f53aaffd640 10 monclient(hunting): dump: 2024-03-25T19:51:01.308 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:01.308 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:01.308 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:01.308 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:01.308 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:01.308 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:01.308 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:01.308 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:01.308 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:01.308 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:01.308 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.307+0000 7f53aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:01.309 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.307+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3344507209 0 0) 0x7f53b4137f10 con 0x7f53b413f710 2024-03-25T19:51:01.309 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.307+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5398004eb0 con 0x7f53b413f710 2024-03-25T19:51:01.309 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.307+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f53a40018b0 con 0x7f53b413f710 2024-03-25T19:51:01.309 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.307+0000 7f53aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:01.309 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.307+0000 7f53aaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:01.309 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.307+0000 7f53aaffd640 10 monclient(hunting): dump: 2024-03-25T19:51:01.309 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:01.309 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:01.309 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:01.309 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:01.309 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:01.309 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:01.309 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:01.309 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:01.309 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:01.310 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:01.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3148899827 0 0) 0x7f53b41dc070 con 0x7f53b40a98f0 2024-03-25T19:51:01.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 >> v1:172.21.15.81:6790/0 conn(0x7f53b413f710 legacy=0x7f53b41db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:01.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 >> v1:172.21.15.98:6789/0 conn(0x7f53b41430c0 legacy=0x7f53b41db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:01.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53aaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:01.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53aaffd640 1 monclient: found mon.a 2024-03-25T19:51:01.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:01.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53aaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:51:01.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b41373e0 con 0x7f53b40a98f0 2024-03-25T19:51:01.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53aaffd640 10 monclient: _finish_auth 0 2024-03-25T19:51:01.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53aaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:01.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:01.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53bc63e640 5 monclient: authenticate success, global_id 4644 2024-03-25T19:51:01.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53bc63e640 10 monclient: _renew_subs 2024-03-25T19:51:01.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53bc63e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:51:01.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53bc63e640 1 -- 172.21.15.81:0/1490330142 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b41358e0 con 0x7f53b40a98f0 2024-03-25T19:51:01.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53bc63e640 10 client.4644.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:01.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53bc63e640 10 monclient: _renew_subs 2024-03-25T19:51:01.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53bc63e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:51:01.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53bc63e640 1 -- 172.21.15.81:0/1490330142 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b4136150 con 0x7f53b40a98f0 2024-03-25T19:51:01.319 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.318+0000 7f53bc63e640 1 librados: init done 2024-03-25T19:51:01.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.319+0000 7f53bc63e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:01.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.319+0000 7f53bc63e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:01.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.319+0000 7f53bc63e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:51:01.320 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.319+0000 7f53bc63e640 1 -- 172.21.15.81:0/1490330142 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b40627b0 con 0x7f53b40a98f0 2024-03-25T19:51:01.609 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.609+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53b00035f0 con 0x7f53b40a98f0 2024-03-25T19:51:01.610 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.609+0000 7f53aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:01.610 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.609+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f53b0003980 con 0x7f53b40a98f0 2024-03-25T19:51:01.610 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.609+0000 7f53aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:01.610 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.610+0000 7f53aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:01.610 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.610+0000 7f53aaffd640 10 monclient: dump: 2024-03-25T19:51:01.610 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:01.610 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:01.610 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:01.610 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:01.610 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:01.610 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:01.610 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:01.610 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:01.610 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:01.611 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:01.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.610+0000 7f53b8bd7640 4 set_mon_vals no callback set 2024-03-25T19:51:01.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.611+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f53b0003ba0 con 0x7f53b40a98f0 2024-03-25T19:51:01.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.611+0000 7f53b9bd9640 10 monclient: get_auth_request con 0x7f539806df70 auth_method 0 2024-03-25T19:51:01.611 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.612+0000 7f53aaffd640 10 client.4644.objecter ms_handle_connect 0x7f539806df70 2024-03-25T19:51:01.617 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.617+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f53b0053600 con 0x7f53b40a98f0 2024-03-25T19:51:01.617 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.617+0000 7f53aaffd640 10 client.4644.objecter ms_dispatch 0x7f53b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:51:01.617 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.617+0000 7f53aaffd640 3 client.4644.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:51:01.618 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.617+0000 7f53aaffd640 3 client.4644.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:51:01.618 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.618+0000 7f53aaffd640 20 client.4644.objecter dump_active .. 0 homeless 2024-03-25T19:51:01.618 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.618+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53b004f110 con 0x7f53b40a98f0 2024-03-25T19:51:01.618 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.618+0000 7f53aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:01.618 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.618+0000 7f53aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:01.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.832+0000 7f53bc63e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:01.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.832+0000 7f53bc63e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:01.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.832+0000 7f53bc63e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:51:01.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.832+0000 7f53bc63e640 1 -- 172.21.15.81:0/1490330142 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53b4135a70 con 0x7f53b40a98f0 2024-03-25T19:51:01.921 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.922+0000 7f53aaffd640 1 -- 172.21.15.81:0/1490330142 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f53b0051040 con 0x7f53b40a98f0 2024-03-25T19:51:01.922 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.922+0000 7f53aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:01.922 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.922+0000 7f53aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:01.922 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:51:01.922 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:01.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.924+0000 7f53bc63e640 10 librados: watch_flush enter 2024-03-25T19:51:01.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.924+0000 7f53bc63e640 10 librados: watch_flush exit 2024-03-25T19:51:01.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.924+0000 7f53bc63e640 20 client.4644.objecter shutdown clearing up homeless session... 2024-03-25T19:51:01.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.924+0000 7f53bc63e640 10 client.4644.objecter successfully canceled tick 2024-03-25T19:51:01.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.925+0000 7f53bc63e640 1 -- 172.21.15.81:0/1490330142 >> v1:172.21.15.98:6800/3980829775 conn(0x7f539806df70 legacy=0x7f5398070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:01.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.925+0000 7f53bc63e640 10 monclient: shutdown 2024-03-25T19:51:01.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.925+0000 7f53bc63e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:01.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.925+0000 7f53bc63e640 1 -- 172.21.15.81:0/1490330142 >> v1:172.21.15.81:6789/0 conn(0x7f53b40a98f0 legacy=0x7f53b4137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:01.925 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.925+0000 7f53bc63e640 1 -- 172.21.15.81:0/1490330142 shutdown_connections 2024-03-25T19:51:01.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.925+0000 7f53bc63e640 1 -- 172.21.15.81:0/1490330142 >> 172.21.15.81:0/1490330142 conn(0x7f53b40b7ea0 msgr2=0x7f53b414ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:01.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.925+0000 7f53bc63e640 1 -- 172.21.15.81:0/1490330142 shutdown_connections 2024-03-25T19:51:01.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.925+0000 7f53bc63e640 1 -- 172.21.15.81:0/1490330142 wait complete. 2024-03-25T19:51:01.926 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:01.925+0000 7f53bc63e640 1 librados: shutdown 2024-03-25T19:51:01.935 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:03.502 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:09:26.144 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:03.518 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:09:26.160 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:03.552 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:09:26.194 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:03.571 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:09:26.213 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:03.576 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:09:26.218 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:03.589 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:26.223 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:03.592 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:26.226 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:03.605 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:09:26.247 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:03.611 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:26.246 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:08.743 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:08.857 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.858+0000 7f9c87edd640 10 monclient: get_monmap_and_config 2024-03-25T19:51:08.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.858+0000 7f9c87edd640 10 monclient: build_initial_monmap 2024-03-25T19:51:08.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.858+0000 7f9c87edd640 10 monclient: monmap: 2024-03-25T19:51:08.858 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:51:08.858 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:08.858 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:51:08.859095+0000 2024-03-25T19:51:08.858 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:51:08.859095+0000 2024-03-25T19:51:08.858 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:08.858 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:08.858 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:51:08.858 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:51:08.858 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:51:08.858 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:08.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.858+0000 7f9c87edd640 1 Processor -- start 2024-03-25T19:51:08.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.859+0000 7f9c87edd640 1 -- start start 2024-03-25T19:51:08.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.859+0000 7f9c87edd640 10 monclient: init 2024-03-25T19:51:08.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.860+0000 7f9c87edd640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:08.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.860+0000 7f9c87edd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:08.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.860+0000 7f9c87edd640 10 monclient(hunting): picked mon.noname-a con 0x7f9c800b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:51:08.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.860+0000 7f9c87edd640 10 monclient(hunting): picked mon.noname-b con 0x7f9c800b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:51:08.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.860+0000 7f9c87edd640 10 monclient(hunting): picked mon.noname-c con 0x7f9c8014be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:51:08.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.860+0000 7f9c87edd640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c800af940 con 0x7f9c800b4d60 2024-03-25T19:51:08.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.860+0000 7f9c87edd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c800b0a70 con 0x7f9c8014be90 2024-03-25T19:51:08.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.860+0000 7f9c87edd640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c800b0d50 con 0x7f9c800b1730 2024-03-25T19:51:08.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.860+0000 7f9c87edd640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:08.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.860+0000 7f9c87edd640 10 monclient(hunting): authenticate will time out at 4503.942267s 2024-03-25T19:51:08.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.864+0000 7f9c85c79640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f9c800b4d60 0x7f9c8013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:39474/0 (socket says 172.21.15.81:39474) 2024-03-25T19:51:08.864 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.864+0000 7f9c85c79640 1 -- 172.21.15.81:0/4062268085 learned_addr learned my addr 172.21.15.81:0/4062268085 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:51:08.873 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.873+0000 7f9c84c77640 1 -- 172.21.15.81:0/4062268085 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374939502 0 0) 0x7f9c800af940 con 0x7f9c800b4d60 2024-03-25T19:51:08.873 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.873+0000 7f9c84c77640 10 monclient(hunting): _negotiate 2024-03-25T19:51:08.873 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.873+0000 7f9c84c77640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:08.873 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.873+0000 7f9c84c77640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:08.873 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.873+0000 7f9c84c77640 10 monclient(hunting): my global_id is 4647 2024-03-25T19:51:08.873 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.873+0000 7f9c84c77640 1 -- 172.21.15.81:0/4062268085 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c6c003520 con 0x7f9c800b4d60 2024-03-25T19:51:08.882 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c84c77640 1 -- 172.21.15.81:0/4062268085 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1646621902 0 0) 0x7f9c6c003520 con 0x7f9c800b4d60 2024-03-25T19:51:08.882 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c84c77640 1 -- 172.21.15.81:0/4062268085 >> v1:172.21.15.81:6790/0 conn(0x7f9c800b1730 legacy=0x7f9c800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:08.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c84c77640 1 -- 172.21.15.81:0/4062268085 >> v1:172.21.15.98:6789/0 conn(0x7f9c8014be90 legacy=0x7f9c8014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:08.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c84c77640 10 monclient: _finish_hunting 0 2024-03-25T19:51:08.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c84c77640 1 monclient: found mon.noname-a 2024-03-25T19:51:08.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c84c77640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:08.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c84c77640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:51:08.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c84c77640 1 -- 172.21.15.81:0/4062268085 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c800b1030 con 0x7f9c800b4d60 2024-03-25T19:51:08.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c84c77640 10 monclient: _finish_auth 0 2024-03-25T19:51:08.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c84c77640 10 monclient: _check_auth_tickets 2024-03-25T19:51:08.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c84c77640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:08.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c84c77640 1 -- 172.21.15.81:0/4062268085 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f9c7c001690 con 0x7f9c800b4d60 2024-03-25T19:51:08.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c87edd640 5 monclient: authenticate success, global_id 4647 2024-03-25T19:51:08.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c87edd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:08.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c84c77640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:08.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c84c77640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:08.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c84c77640 10 monclient: dump: 2024-03-25T19:51:08.883 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:08.884 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:08.884 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:08.884 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:08.884 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:08.884 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:08.884 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:08.884 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:08.884 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:08.884 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:08.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.883+0000 7f9c87edd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:08.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.894+0000 7f9c84c77640 1 -- 172.21.15.81:0/4062268085 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c7c0017f0 con 0x7f9c800b4d60 2024-03-25T19:51:08.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.894+0000 7f9c84c77640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:08.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.894+0000 7f9c84c77640 1 -- 172.21.15.81:0/4062268085 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f9c7c001c00 con 0x7f9c800b4d60 2024-03-25T19:51:08.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.894+0000 7f9c84c77640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:08.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.894+0000 7f9c84c77640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:08.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.894+0000 7f9c84c77640 10 monclient: dump: 2024-03-25T19:51:08.894 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:08.894 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:08.894 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:08.894 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:08.894 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:08.894 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:08.895 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:08.895 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:08.895 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:08.895 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:08.895 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.894+0000 7f9c87edd640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:08.895 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.894+0000 7f9c87edd640 4 set_mon_vals no callback set 2024-03-25T19:51:08.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.895+0000 7f9c87edd640 10 monclient: shutdown 2024-03-25T19:51:08.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.895+0000 7f9c87edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:08.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.895+0000 7f9c87edd640 1 -- 172.21.15.81:0/4062268085 >> v1:172.21.15.81:6789/0 conn(0x7f9c800b4d60 legacy=0x7f9c8013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:08.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.895+0000 7f9c87edd640 1 -- 172.21.15.81:0/4062268085 shutdown_connections 2024-03-25T19:51:08.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.895+0000 7f9c87edd640 1 -- 172.21.15.81:0/4062268085 >> 172.21.15.81:0/4062268085 conn(0x7f9c800617a0 msgr2=0x7f9c80061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:08.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.895+0000 7f9c87edd640 1 -- 172.21.15.81:0/4062268085 shutdown_connections 2024-03-25T19:51:08.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.895+0000 7f9c87edd640 1 -- 172.21.15.81:0/4062268085 wait complete. 2024-03-25T19:51:08.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.896+0000 7f9c87edd640 10 monclient: build_initial_monmap 2024-03-25T19:51:08.897 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.896+0000 7f9c87edd640 10 monclient: monmap: 2024-03-25T19:51:08.897 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:51:08.897 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:08.898 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:51:08.898 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:51:08.898 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:08.898 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:51:08.898 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:51:08.898 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:51:08.898 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:51:08.898 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:08.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.896+0000 7f9c87edd640 1 librados: starting msgr at 2024-03-25T19:51:08.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.896+0000 7f9c87edd640 1 librados: starting objecter 2024-03-25T19:51:08.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.896+0000 7f9c87edd640 1 Processor -- start 2024-03-25T19:51:08.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.896+0000 7f9c87edd640 1 -- start start 2024-03-25T19:51:08.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.896+0000 7f9c87edd640 1 librados: setting wanted keys 2024-03-25T19:51:08.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.896+0000 7f9c87edd640 1 librados: calling monclient init 2024-03-25T19:51:08.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.896+0000 7f9c87edd640 10 monclient: init 2024-03-25T19:51:08.898 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.897+0000 7f9c87edd640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:08.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.897+0000 7f9c87edd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:08.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.897+0000 7f9c87edd640 10 monclient(hunting): picked mon.noname-c con 0x7f9c800b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:51:08.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.897+0000 7f9c87edd640 10 monclient(hunting): picked mon.noname-a con 0x7f9c800b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:51:08.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.897+0000 7f9c87edd640 10 monclient(hunting): picked mon.noname-b con 0x7f9c8014be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:51:08.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.897+0000 7f9c87edd640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c8013a940 con 0x7f9c800b4d60 2024-03-25T19:51:08.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.897+0000 7f9c87edd640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c8013ab90 con 0x7f9c8014be90 2024-03-25T19:51:08.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.897+0000 7f9c87edd640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c801d7840 con 0x7f9c800b1730 2024-03-25T19:51:08.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.897+0000 7f9c87edd640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:08.899 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.897+0000 7f9c87edd640 10 monclient(hunting): authenticate will time out at 4503.979186s 2024-03-25T19:51:08.913 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.913+0000 7f9c8647a640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f9c8014be90 0x7f9c801e3030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:51758/0 (socket says 172.21.15.81:51758) 2024-03-25T19:51:08.913 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.913+0000 7f9c8647a640 1 -- 172.21.15.81:0/2500064066 learned_addr learned my addr 172.21.15.81:0/2500064066 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:51:08.915 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.916+0000 7f9c767fc640 10 client.?.objecter ms_handle_connect 0x7f9c8014be90 2024-03-25T19:51:08.915 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.916+0000 7f9c767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:08.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.916+0000 7f9c767fc640 10 client.?.objecter ms_handle_connect 0x7f9c800b4d60 2024-03-25T19:51:08.916 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.916+0000 7f9c767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:08.923 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.923+0000 7f9c767fc640 10 client.?.objecter ms_handle_connect 0x7f9c800b1730 2024-03-25T19:51:08.923 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.923+0000 7f9c767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:08.929 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.929+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148173651 0 0) 0x7f9c8013a940 con 0x7f9c800b4d60 2024-03-25T19:51:08.929 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.929+0000 7f9c767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:08.929 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.929+0000 7f9c767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:08.929 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.929+0000 7f9c767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:08.929 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.929+0000 7f9c767fc640 10 monclient(hunting): my global_id is 4650 2024-03-25T19:51:08.929 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.929+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c64004eb0 con 0x7f9c800b4d60 2024-03-25T19:51:08.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.932+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1625874560 0 0) 0x7f9c8013ab90 con 0x7f9c8014be90 2024-03-25T19:51:08.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.932+0000 7f9c767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:08.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.932+0000 7f9c767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:08.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.932+0000 7f9c767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:08.931 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.932+0000 7f9c767fc640 10 monclient(hunting): my global_id is 4636 2024-03-25T19:51:08.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.932+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c8013a940 con 0x7f9c8014be90 2024-03-25T19:51:08.948 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.948+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2141883826 0 0) 0x7f9c8013a940 con 0x7f9c8014be90 2024-03-25T19:51:08.948 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.948+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c8013ab90 con 0x7f9c8014be90 2024-03-25T19:51:08.948 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.948+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f9c78001c10 con 0x7f9c8014be90 2024-03-25T19:51:08.948 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.948+0000 7f9c767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:08.948 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.948+0000 7f9c767fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:08.948 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.948+0000 7f9c767fc640 10 monclient(hunting): dump: 2024-03-25T19:51:08.949 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:08.949 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:08.949 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:08.949 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:08.949 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:08.949 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:08.949 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:08.949 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:08.949 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:08.949 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:08.949 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.948+0000 7f9c767fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:08.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.953+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270570632 0 0) 0x7f9c801d7840 con 0x7f9c800b1730 2024-03-25T19:51:08.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.953+0000 7f9c767fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:08.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.953+0000 7f9c767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:08.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.953+0000 7f9c767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:08.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.953+0000 7f9c767fc640 10 monclient(hunting): my global_id is 4616 2024-03-25T19:51:08.953 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.953+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c8013a940 con 0x7f9c800b1730 2024-03-25T19:51:08.959 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.959+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3511550506 0 0) 0x7f9c64004eb0 con 0x7f9c800b4d60 2024-03-25T19:51:08.959 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.959+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c801d7840 con 0x7f9c800b4d60 2024-03-25T19:51:08.959 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.959+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f9c70001930 con 0x7f9c800b4d60 2024-03-25T19:51:08.959 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.959+0000 7f9c767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:08.959 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.959+0000 7f9c767fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:08.959 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.959+0000 7f9c767fc640 10 monclient(hunting): dump: 2024-03-25T19:51:08.959 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:08.959 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:08.959 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:08.959 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:08.959 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:08.959 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:08.960 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:08.960 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:08.960 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:08.960 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:08.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 814169793 0 0) 0x7f9c8013ab90 con 0x7f9c8014be90 2024-03-25T19:51:08.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 >> v1:172.21.15.81:6790/0 conn(0x7f9c800b1730 legacy=0x7f9c8013bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:08.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 >> v1:172.21.15.81:6789/0 conn(0x7f9c800b4d60 legacy=0x7f9c801d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:08.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c767fc640 10 monclient: _finish_hunting 0 2024-03-25T19:51:08.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c767fc640 1 monclient: found mon.b 2024-03-25T19:51:08.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:08.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c767fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:51:08.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8013c240 con 0x7f9c8014be90 2024-03-25T19:51:08.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c767fc640 10 monclient: _finish_auth 0 2024-03-25T19:51:08.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c767fc640 10 monclient: _check_auth_tickets 2024-03-25T19:51:08.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:08.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c87edd640 5 monclient: authenticate success, global_id 4636 2024-03-25T19:51:08.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c87edd640 10 monclient: _renew_subs 2024-03-25T19:51:08.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c87edd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:51:08.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c87edd640 1 -- 172.21.15.81:0/2500064066 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c80139ce0 con 0x7f9c8014be90 2024-03-25T19:51:08.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c87edd640 10 client.4636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:08.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c87edd640 10 monclient: _renew_subs 2024-03-25T19:51:08.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c87edd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:51:08.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c87edd640 1 -- 172.21.15.81:0/2500064066 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c801d8650 con 0x7f9c8014be90 2024-03-25T19:51:08.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.976+0000 7f9c87edd640 1 librados: init done 2024-03-25T19:51:08.978 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.977+0000 7f9c87edd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:08.978 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.977+0000 7f9c87edd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:08.978 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.977+0000 7f9c87edd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:51:08.978 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:08.977+0000 7f9c87edd640 1 -- 172.21.15.81:0/2500064066 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c801401f0 con 0x7f9c8014be90 2024-03-25T19:51:09.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.261+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c78003990 con 0x7f9c8014be90 2024-03-25T19:51:09.261 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.261+0000 7f9c767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:09.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.261+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f9c78002100 con 0x7f9c8014be90 2024-03-25T19:51:09.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.261+0000 7f9c767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:09.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.261+0000 7f9c767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:09.262 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.261+0000 7f9c767fc640 10 monclient: dump: 2024-03-25T19:51:09.262 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:09.262 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:09.262 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:09.262 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:09.262 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:09.262 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:09.262 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:09.262 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:09.262 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:09.262 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:09.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.261+0000 7f9c77fff640 4 set_mon_vals no callback set 2024-03-25T19:51:09.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.261+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f9c78003de0 con 0x7f9c8014be90 2024-03-25T19:51:09.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.262+0000 7f9c85c79640 10 monclient: get_auth_request con 0x7f9c6406df70 auth_method 0 2024-03-25T19:51:09.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.262+0000 7f9c767fc640 10 client.4636.objecter ms_handle_connect 0x7f9c6406df70 2024-03-25T19:51:09.267 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.267+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f9c780047e0 con 0x7f9c8014be90 2024-03-25T19:51:09.268 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.267+0000 7f9c767fc640 10 client.4636.objecter ms_dispatch 0x7f9c80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:51:09.268 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.267+0000 7f9c767fc640 3 client.4636.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:51:09.268 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.267+0000 7f9c767fc640 3 client.4636.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:51:09.268 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.267+0000 7f9c767fc640 20 client.4636.objecter dump_active .. 0 homeless 2024-03-25T19:51:09.268 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.268+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c780503a0 con 0x7f9c8014be90 2024-03-25T19:51:09.268 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.268+0000 7f9c767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:09.268 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.268+0000 7f9c767fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:09.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.508+0000 7f9c87edd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:09.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.508+0000 7f9c87edd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:09.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.508+0000 7f9c87edd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:51:09.509 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.508+0000 7f9c87edd640 1 -- 172.21.15.81:0/2500064066 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c800627b0 con 0x7f9c8014be90 2024-03-25T19:51:09.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.564+0000 7f9c767fc640 1 -- 172.21.15.81:0/2500064066 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f9c780524a0 con 0x7f9c8014be90 2024-03-25T19:51:09.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.564+0000 7f9c767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:09.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.564+0000 7f9c767fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:09.565 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:51:09.565 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:09.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.567+0000 7f9c87edd640 10 librados: watch_flush enter 2024-03-25T19:51:09.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.567+0000 7f9c87edd640 10 librados: watch_flush exit 2024-03-25T19:51:09.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.567+0000 7f9c87edd640 20 client.4636.objecter shutdown clearing up homeless session... 2024-03-25T19:51:09.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.567+0000 7f9c87edd640 10 client.4636.objecter successfully canceled tick 2024-03-25T19:51:09.568 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.567+0000 7f9c87edd640 1 -- 172.21.15.81:0/2500064066 >> v1:172.21.15.98:6800/3980829775 conn(0x7f9c6406df70 legacy=0x7f9c64070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.567+0000 7f9c87edd640 10 monclient: shutdown 2024-03-25T19:51:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.567+0000 7f9c87edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.567+0000 7f9c87edd640 1 -- 172.21.15.81:0/2500064066 >> v1:172.21.15.98:6789/0 conn(0x7f9c8014be90 legacy=0x7f9c801e3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.567+0000 7f9c87edd640 1 -- 172.21.15.81:0/2500064066 shutdown_connections 2024-03-25T19:51:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.567+0000 7f9c87edd640 1 -- 172.21.15.81:0/2500064066 >> 172.21.15.81:0/2500064066 conn(0x7f9c800617a0 msgr2=0x7f9c80137510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.567+0000 7f9c87edd640 1 -- 172.21.15.81:0/2500064066 shutdown_connections 2024-03-25T19:51:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.567+0000 7f9c87edd640 1 -- 172.21.15.81:0/2500064066 wait complete. 2024-03-25T19:51:09.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:09.567+0000 7f9c87edd640 1 librados: shutdown 2024-03-25T19:51:09.581 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:16.387 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:16.498 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.499+0000 7facb3088640 10 monclient: get_monmap_and_config 2024-03-25T19:51:16.499 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.499+0000 7facb3088640 10 monclient: build_initial_monmap 2024-03-25T19:51:16.499 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.499+0000 7facb3088640 10 monclient: monmap: 2024-03-25T19:51:16.499 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:51:16.499 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:16.499 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:51:16.500026+0000 2024-03-25T19:51:16.499 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:51:16.500026+0000 2024-03-25T19:51:16.499 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:16.499 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:16.499 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:51:16.499 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:51:16.499 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:51:16.499 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:16.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.499+0000 7facb3088640 1 Processor -- start 2024-03-25T19:51:16.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.500+0000 7facb3088640 1 -- start start 2024-03-25T19:51:16.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.500+0000 7facb3088640 10 monclient: init 2024-03-25T19:51:16.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.501+0000 7facb3088640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:16.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.501+0000 7facb3088640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:51:16.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.501+0000 7facb3088640 10 monclient(hunting): picked mon.noname-c con 0x7facac0b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:51:16.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.501+0000 7facb3088640 10 monclient(hunting): picked mon.noname-b con 0x7facac0b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:51:16.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.501+0000 7facb3088640 10 monclient(hunting): picked mon.noname-a con 0x7facac14be90 addr v1:172.21.15.81:6789/0 2024-03-25T19:51:16.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.501+0000 7facb3088640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facac0af940 con 0x7facac14be90 2024-03-25T19:51:16.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.501+0000 7facb3088640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facac0b0a70 con 0x7facac0b4d60 2024-03-25T19:51:16.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.501+0000 7facb3088640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facac0b0d50 con 0x7facac0b1730 2024-03-25T19:51:16.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.501+0000 7facb3088640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:16.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.501+0000 7facb3088640 10 monclient(hunting): authenticate will time out at 4511.583119s 2024-03-25T19:51:16.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.504+0000 7facb0e24640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7facac0b4d60 0x7facac13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:56830/0 (socket says 172.21.15.81:56830) 2024-03-25T19:51:16.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.504+0000 7facb0e24640 1 -- 172.21.15.81:0/2914653933 learned_addr learned my addr 172.21.15.81:0/2914653933 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:51:16.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.560+0000 7facab7fe640 1 -- 172.21.15.81:0/2914653933 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015757179 0 0) 0x7facac0b0d50 con 0x7facac0b1730 2024-03-25T19:51:16.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.560+0000 7facab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:16.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.560+0000 7facab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:16.560 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.560+0000 7facab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:16.560 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.560+0000 7facab7fe640 10 monclient(hunting): my global_id is 4619 2024-03-25T19:51:16.560 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.560+0000 7facab7fe640 1 -- 172.21.15.81:0/2914653933 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac9c003520 con 0x7facac0b1730 2024-03-25T19:51:16.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.569+0000 7facab7fe640 1 -- 172.21.15.81:0/2914653933 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127488570 0 0) 0x7facac0b0a70 con 0x7facac0b4d60 2024-03-25T19:51:16.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.569+0000 7facab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:16.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.569+0000 7facab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:16.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.569+0000 7facab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:16.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.569+0000 7facab7fe640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:51:16.569 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.569+0000 7facab7fe640 1 -- 172.21.15.81:0/2914653933 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facac0b0d50 con 0x7facac0b4d60 2024-03-25T19:51:16.581 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.581+0000 7facab7fe640 1 -- 172.21.15.81:0/2914653933 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2606918559 0 0) 0x7facac0b0d50 con 0x7facac0b4d60 2024-03-25T19:51:16.581 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.581+0000 7facab7fe640 1 -- 172.21.15.81:0/2914653933 >> v1:172.21.15.81:6790/0 conn(0x7facac0b1730 legacy=0x7facac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:16.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.582+0000 7facab7fe640 1 -- 172.21.15.81:0/2914653933 >> v1:172.21.15.81:6789/0 conn(0x7facac14be90 legacy=0x7facac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:16.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.582+0000 7facab7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:16.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.582+0000 7facab7fe640 1 monclient: found mon.noname-c 2024-03-25T19:51:16.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.582+0000 7facab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:16.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.582+0000 7facab7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:51:16.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.582+0000 7facab7fe640 1 -- 172.21.15.81:0/2914653933 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac0b1030 con 0x7facac0b4d60 2024-03-25T19:51:16.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.582+0000 7facab7fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:16.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.582+0000 7facab7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:16.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.582+0000 7facab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:16.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.582+0000 7facab7fe640 1 -- 172.21.15.81:0/2914653933 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7faca0001670 con 0x7facac0b4d60 2024-03-25T19:51:16.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.582+0000 7facab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:16.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.582+0000 7facab7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:16.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.582+0000 7facab7fe640 10 monclient: dump: 2024-03-25T19:51:16.582 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:16.582 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:16.583 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:16.583 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:16.583 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:16.583 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:16.583 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:16.583 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:16.583 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:16.583 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:16.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.582+0000 7facb3088640 5 monclient: authenticate success, global_id 4639 2024-03-25T19:51:16.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.582+0000 7facb3088640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:16.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.585+0000 7facab7fe640 1 -- 172.21.15.81:0/2914653933 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faca00017d0 con 0x7facac0b4d60 2024-03-25T19:51:16.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.585+0000 7facab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:16.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.585+0000 7facab7fe640 1 -- 172.21.15.81:0/2914653933 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7faca0001ba0 con 0x7facac0b4d60 2024-03-25T19:51:16.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.585+0000 7facab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:16.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.585+0000 7facab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:16.585 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.585+0000 7facab7fe640 10 monclient: dump: 2024-03-25T19:51:16.585 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:16.585 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:16.585 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:16.585 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:16.585 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:16.585 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:16.585 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:16.585 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:16.586 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:16.586 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:16.586 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.585+0000 7facb3088640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:16.586 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.585+0000 7facb3088640 4 set_mon_vals no callback set 2024-03-25T19:51:16.587 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.586+0000 7facb3088640 10 monclient: shutdown 2024-03-25T19:51:16.587 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.586+0000 7facb3088640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:16.587 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.586+0000 7facb3088640 1 -- 172.21.15.81:0/2914653933 >> v1:172.21.15.98:6789/0 conn(0x7facac0b4d60 legacy=0x7facac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:16.587 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.586+0000 7facb3088640 1 -- 172.21.15.81:0/2914653933 shutdown_connections 2024-03-25T19:51:16.588 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.586+0000 7facb3088640 1 -- 172.21.15.81:0/2914653933 >> 172.21.15.81:0/2914653933 conn(0x7facac0617a0 msgr2=0x7facac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:16.588 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.586+0000 7facb3088640 1 -- 172.21.15.81:0/2914653933 shutdown_connections 2024-03-25T19:51:16.588 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.586+0000 7facb3088640 1 -- 172.21.15.81:0/2914653933 wait complete. 2024-03-25T19:51:16.588 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.587+0000 7facb3088640 10 monclient: build_initial_monmap 2024-03-25T19:51:16.588 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.587+0000 7facb3088640 10 monclient: monmap: 2024-03-25T19:51:16.588 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:51:16.588 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:16.588 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:51:16.588 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:51:16.588 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:16.588 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:51:16.588 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:51:16.588 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:51:16.588 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:51:16.588 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:16.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.587+0000 7facb3088640 1 librados: starting msgr at 2024-03-25T19:51:16.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.587+0000 7facb3088640 1 librados: starting objecter 2024-03-25T19:51:16.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.587+0000 7facb3088640 1 Processor -- start 2024-03-25T19:51:16.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.587+0000 7facb3088640 1 -- start start 2024-03-25T19:51:16.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.587+0000 7facb3088640 1 librados: setting wanted keys 2024-03-25T19:51:16.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.587+0000 7facb3088640 1 librados: calling monclient init 2024-03-25T19:51:16.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.587+0000 7facb3088640 10 monclient: init 2024-03-25T19:51:16.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.588+0000 7facb3088640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:16.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.588+0000 7facb3088640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:51:16.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.588+0000 7facb3088640 10 monclient(hunting): picked mon.noname-c con 0x7facac0b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:51:16.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.588+0000 7facb3088640 10 monclient(hunting): picked mon.noname-b con 0x7facac0b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:51:16.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.588+0000 7facb3088640 10 monclient(hunting): picked mon.noname-a con 0x7facac14be90 addr v1:172.21.15.81:6789/0 2024-03-25T19:51:16.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.588+0000 7facb3088640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facac13a650 con 0x7facac14be90 2024-03-25T19:51:16.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.588+0000 7facb3088640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facac13b870 con 0x7facac0b4d60 2024-03-25T19:51:16.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.588+0000 7facb3088640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facac142170 con 0x7facac0b1730 2024-03-25T19:51:16.590 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.588+0000 7facb3088640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:16.591 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.588+0000 7facb3088640 10 monclient(hunting): authenticate will time out at 4511.670592s 2024-03-25T19:51:16.591 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.592+0000 7facb0e24640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7facac0b1730 0x7facac142840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:41522/0 (socket says 172.21.15.81:41522) 2024-03-25T19:51:16.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.592+0000 7facb0e24640 1 -- 172.21.15.81:0/1785394898 learned_addr learned my addr 172.21.15.81:0/1785394898 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:51:16.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.593+0000 7faca97fa640 10 client.?.objecter ms_handle_connect 0x7facac0b1730 2024-03-25T19:51:16.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.593+0000 7faca97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:16.598 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.598+0000 7faca97fa640 10 client.?.objecter ms_handle_connect 0x7facac0b4d60 2024-03-25T19:51:16.598 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.598+0000 7faca97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:16.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.608+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008086514 0 0) 0x7facac142170 con 0x7facac0b1730 2024-03-25T19:51:16.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.608+0000 7faca97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:16.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.608+0000 7faca97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:16.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.608+0000 7faca97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:16.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.608+0000 7faca97fa640 10 monclient(hunting): my global_id is 4622 2024-03-25T19:51:16.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.608+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac90004eb0 con 0x7facac0b1730 2024-03-25T19:51:16.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.608+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357828234 0 0) 0x7facac13b870 con 0x7facac0b4d60 2024-03-25T19:51:16.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.608+0000 7faca97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:16.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.608+0000 7faca97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:16.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.608+0000 7faca97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:16.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.608+0000 7faca97fa640 10 monclient(hunting): my global_id is 4642 2024-03-25T19:51:16.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.608+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facac142170 con 0x7facac0b4d60 2024-03-25T19:51:16.612 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.613+0000 7faca97fa640 10 client.?.objecter ms_handle_connect 0x7facac14be90 2024-03-25T19:51:16.612 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.613+0000 7faca97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:16.618 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.618+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 710699658 0 0) 0x7fac90004eb0 con 0x7facac0b1730 2024-03-25T19:51:16.618 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.618+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facac13b870 con 0x7facac0b1730 2024-03-25T19:51:16.618 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.618+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7faca0001ba0 con 0x7facac0b1730 2024-03-25T19:51:16.618 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.618+0000 7faca97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:16.618 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.618+0000 7faca97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:16.618 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.618+0000 7faca97fa640 10 monclient(hunting): dump: 2024-03-25T19:51:16.618 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:16.618 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:16.618 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:16.618 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:16.618 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:16.618 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:16.619 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:16.619 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:16.619 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:16.619 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:16.619 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.618+0000 7faca97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:16.623 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.623+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1384851401 0 0) 0x7facac142170 con 0x7facac0b4d60 2024-03-25T19:51:16.623 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.623+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac90004eb0 con 0x7facac0b4d60 2024-03-25T19:51:16.623 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.623+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fac98001950 con 0x7facac0b4d60 2024-03-25T19:51:16.623 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.623+0000 7faca97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:16.623 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.623+0000 7faca97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:16.623 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.623+0000 7faca97fa640 10 monclient(hunting): dump: 2024-03-25T19:51:16.623 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:16.623 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:16.623 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:16.623 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:16.623 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:16.623 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:16.623 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:16.624 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:16.624 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:16.624 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:16.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3122754452 0 0) 0x7facac13b870 con 0x7facac0b1730 2024-03-25T19:51:16.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 >> v1:172.21.15.98:6789/0 conn(0x7facac0b4d60 legacy=0x7facac1d7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:16.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 >> v1:172.21.15.81:6789/0 conn(0x7facac14be90 legacy=0x7facac1d8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:16.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7faca97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:51:16.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7faca97fa640 1 monclient: found mon.c 2024-03-25T19:51:16.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7faca97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:16.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7faca97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:51:16.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac142300 con 0x7facac0b1730 2024-03-25T19:51:16.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7faca97fa640 10 monclient: _finish_auth 0 2024-03-25T19:51:16.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7faca97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:51:16.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7faca97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:16.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7facb3088640 5 monclient: authenticate success, global_id 4622 2024-03-25T19:51:16.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7facb3088640 10 monclient: _renew_subs 2024-03-25T19:51:16.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7facb3088640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:51:16.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7facb3088640 1 -- 172.21.15.81:0/1785394898 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facac140960 con 0x7facac0b1730 2024-03-25T19:51:16.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7facb3088640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:16.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7facb3088640 10 monclient: _renew_subs 2024-03-25T19:51:16.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7facb3088640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:51:16.629 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7facb3088640 1 -- 172.21.15.81:0/1785394898 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7facac140380 con 0x7facac0b1730 2024-03-25T19:51:16.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.628+0000 7facb3088640 1 librados: init done 2024-03-25T19:51:16.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.629+0000 7facb3088640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:16.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.629+0000 7facb3088640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:16.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.629+0000 7facb3088640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:51:16.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.629+0000 7facb3088640 1 -- 172.21.15.81:0/1785394898 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facac141b00 con 0x7facac0b1730 2024-03-25T19:51:16.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.932+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faca0004a90 con 0x7facac0b1730 2024-03-25T19:51:16.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.932+0000 7faca97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:16.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.932+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7faca0004ea0 con 0x7facac0b1730 2024-03-25T19:51:16.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.932+0000 7faca97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:16.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.932+0000 7faca97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:16.932 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.932+0000 7faca97fa640 10 monclient: dump: 2024-03-25T19:51:16.932 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:16.932 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:16.933 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:16.933 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:16.933 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:16.933 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:16.933 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:16.933 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:16.933 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:16.933 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:16.933 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.932+0000 7facaaffd640 4 set_mon_vals no callback set 2024-03-25T19:51:16.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.933+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7faca00050c0 con 0x7facac0b1730 2024-03-25T19:51:16.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.934+0000 7facabfff640 10 monclient: get_auth_request con 0x7fac9006de90 auth_method 0 2024-03-25T19:51:16.934 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.934+0000 7faca97fa640 10 client.4622.objecter ms_handle_connect 0x7fac9006de90 2024-03-25T19:51:16.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.939+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7faca00549b0 con 0x7facac0b1730 2024-03-25T19:51:16.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.939+0000 7faca97fa640 10 client.4622.objecter ms_dispatch 0x7facac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:51:16.939 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.939+0000 7faca97fa640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:51:16.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.939+0000 7faca97fa640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:51:16.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.940+0000 7faca97fa640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-25T19:51:16.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.940+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faca00504c0 con 0x7facac0b1730 2024-03-25T19:51:16.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.940+0000 7faca97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:16.940 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:16.940+0000 7faca97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:17.194 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.194+0000 7facb3088640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:17.194 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.194+0000 7facb3088640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:17.194 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.194+0000 7facb3088640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:51:17.194 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.194+0000 7facb3088640 1 -- 172.21.15.81:0/1785394898 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facac0627b0 con 0x7facac0b1730 2024-03-25T19:51:17.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.231+0000 7faca97fa640 1 -- 172.21.15.81:0/1785394898 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7faca00523f0 con 0x7facac0b1730 2024-03-25T19:51:17.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.231+0000 7faca97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:17.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.231+0000 7faca97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:17.231 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:51:17.231 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:17.233 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.233+0000 7facb3088640 10 librados: watch_flush enter 2024-03-25T19:51:17.233 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.233+0000 7facb3088640 10 librados: watch_flush exit 2024-03-25T19:51:17.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.233+0000 7facb3088640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-25T19:51:17.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.233+0000 7facb3088640 10 client.4622.objecter successfully canceled tick 2024-03-25T19:51:17.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.233+0000 7facb3088640 1 -- 172.21.15.81:0/1785394898 >> v1:172.21.15.98:6800/3980829775 conn(0x7fac9006de90 legacy=0x7fac90070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:17.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.233+0000 7facb3088640 10 monclient: shutdown 2024-03-25T19:51:17.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.233+0000 7facb3088640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:17.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.233+0000 7facb3088640 1 -- 172.21.15.81:0/1785394898 >> v1:172.21.15.81:6790/0 conn(0x7facac0b1730 legacy=0x7facac142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:17.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.233+0000 7facb3088640 1 -- 172.21.15.81:0/1785394898 shutdown_connections 2024-03-25T19:51:17.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.233+0000 7facb3088640 1 -- 172.21.15.81:0/1785394898 >> 172.21.15.81:0/1785394898 conn(0x7facac0617a0 msgr2=0x7facac0b2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:17.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.234+0000 7facb3088640 1 -- 172.21.15.81:0/1785394898 shutdown_connections 2024-03-25T19:51:17.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.234+0000 7facb3088640 1 -- 172.21.15.81:0/1785394898 wait complete. 2024-03-25T19:51:17.234 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:17.234+0000 7facb3088640 1 librados: shutdown 2024-03-25T19:51:17.247 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:23.590 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:09:46.232 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:23.649 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:09:46.290 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:23.675 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:09:46.317 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:23.700 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:09:46.342 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:23.700 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:46.335 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:23.707 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:09:46.349 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:23.710 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:46.344 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:23.733 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:09:46.368 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:23.753 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:09:46.395 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:24.053 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:24.164 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.165+0000 7f33d1ac2640 10 monclient: get_monmap_and_config 2024-03-25T19:51:24.164 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.165+0000 7f33d1ac2640 10 monclient: build_initial_monmap 2024-03-25T19:51:24.165 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.165+0000 7f33d1ac2640 10 monclient: monmap: 2024-03-25T19:51:24.165 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:51:24.165 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:24.165 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:51:24.165974+0000 2024-03-25T19:51:24.165 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:51:24.165974+0000 2024-03-25T19:51:24.165 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:24.165 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:24.165 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:51:24.165 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:51:24.165 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:51:24.165 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:24.166 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.166+0000 7f33d1ac2640 1 Processor -- start 2024-03-25T19:51:24.166 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.166+0000 7f33d1ac2640 1 -- start start 2024-03-25T19:51:24.166 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.166+0000 7f33d1ac2640 10 monclient: init 2024-03-25T19:51:24.167 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.167+0000 7f33d1ac2640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:24.167 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.167+0000 7f33d1ac2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:24.167 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.167+0000 7f33d1ac2640 10 monclient(hunting): picked mon.noname-a con 0x7f33cc0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:51:24.167 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.167+0000 7f33d1ac2640 10 monclient(hunting): picked mon.noname-b con 0x7f33cc13f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:51:24.167 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.167+0000 7f33d1ac2640 10 monclient(hunting): picked mon.noname-c con 0x7f33cc1430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:51:24.167 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.167+0000 7f33d1ac2640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc142520 con 0x7f33cc0a98f0 2024-03-25T19:51:24.167 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.167+0000 7f33d1ac2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc0b28a0 con 0x7f33cc1430c0 2024-03-25T19:51:24.167 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.167+0000 7f33d1ac2640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc0b2a30 con 0x7f33cc13f710 2024-03-25T19:51:24.167 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.167+0000 7f33d1ac2640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:24.167 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.167+0000 7f33d1ac2640 10 monclient(hunting): authenticate will time out at 4519.249020s 2024-03-25T19:51:24.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.170+0000 7f33cb7fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f33cc0a98f0 0x7f33cc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:46448/0 (socket says 172.21.15.81:46448) 2024-03-25T19:51:24.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.170+0000 7f33cb7fe640 1 -- 172.21.15.81:0/2750535615 learned_addr learned my addr 172.21.15.81:0/2750535615 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:51:24.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.174+0000 7f33ca7fc640 1 -- 172.21.15.81:0/2750535615 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078649997 0 0) 0x7f33cc142520 con 0x7f33cc0a98f0 2024-03-25T19:51:24.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.174+0000 7f33ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:24.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.174+0000 7f33ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:24.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.174+0000 7f33ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:24.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.174+0000 7f33ca7fc640 10 monclient(hunting): my global_id is 4659 2024-03-25T19:51:24.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.174+0000 7f33ca7fc640 1 -- 172.21.15.81:0/2750535615 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33b4003520 con 0x7f33cc0a98f0 2024-03-25T19:51:24.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33ca7fc640 1 -- 172.21.15.81:0/2750535615 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 123633443 0 0) 0x7f33b4003520 con 0x7f33cc0a98f0 2024-03-25T19:51:24.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33ca7fc640 1 -- 172.21.15.81:0/2750535615 >> v1:172.21.15.81:6790/0 conn(0x7f33cc13f710 legacy=0x7f33cc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:24.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33ca7fc640 1 -- 172.21.15.81:0/2750535615 >> v1:172.21.15.98:6789/0 conn(0x7f33cc1430c0 legacy=0x7f33cc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:51:24.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33ca7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:51:24.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33ca7fc640 1 monclient: found mon.noname-a 2024-03-25T19:51:24.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:24.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33ca7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:51:24.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33ca7fc640 1 -- 172.21.15.81:0/2750535615 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc0b44c0 con 0x7f33cc0a98f0 2024-03-25T19:51:24.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33ca7fc640 10 monclient: _finish_auth 0 2024-03-25T19:51:24.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33ca7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:51:24.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:24.184 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33ca7fc640 1 -- 172.21.15.81:0/2750535615 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f33bc001690 con 0x7f33cc0a98f0 2024-03-25T19:51:24.185 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33d1ac2640 5 monclient: authenticate success, global_id 4659 2024-03-25T19:51:24.185 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33d1ac2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:24.185 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:24.185 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33ca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:24.185 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33ca7fc640 10 monclient: dump: 2024-03-25T19:51:24.185 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:24.185 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:24.185 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:24.185 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:24.185 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:24.185 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:24.185 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:24.185 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:24.185 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:24.185 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:24.186 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.184+0000 7f33d1ac2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:24.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.196+0000 7f33ca7fc640 1 -- 172.21.15.81:0/2750535615 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33bc001860 con 0x7f33cc0a98f0 2024-03-25T19:51:24.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.196+0000 7f33ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:24.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.196+0000 7f33ca7fc640 1 -- 172.21.15.81:0/2750535615 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f33bc001cb0 con 0x7f33cc0a98f0 2024-03-25T19:51:24.195 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.196+0000 7f33ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:24.196 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.196+0000 7f33ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:24.196 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.196+0000 7f33ca7fc640 10 monclient: dump: 2024-03-25T19:51:24.196 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:24.196 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:24.196 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:24.196 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:24.196 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:24.196 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:24.196 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:24.196 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:24.196 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:24.196 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:24.196 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.196+0000 7f33d1ac2640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:24.196 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.196+0000 7f33d1ac2640 4 set_mon_vals no callback set 2024-03-25T19:51:24.199 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.196+0000 7f33d1ac2640 10 monclient: shutdown 2024-03-25T19:51:24.199 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.196+0000 7f33d1ac2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:24.199 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.196+0000 7f33d1ac2640 1 -- 172.21.15.81:0/2750535615 >> v1:172.21.15.81:6789/0 conn(0x7f33cc0a98f0 legacy=0x7f33cc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:24.199 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.196+0000 7f33d1ac2640 1 -- 172.21.15.81:0/2750535615 shutdown_connections 2024-03-25T19:51:24.199 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.196+0000 7f33d1ac2640 1 -- 172.21.15.81:0/2750535615 >> 172.21.15.81:0/2750535615 conn(0x7f33cc0b7ea0 msgr2=0x7f33cc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:24.199 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.196+0000 7f33d1ac2640 1 -- 172.21.15.81:0/2750535615 shutdown_connections 2024-03-25T19:51:24.199 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.197+0000 7f33d1ac2640 1 -- 172.21.15.81:0/2750535615 wait complete. 2024-03-25T19:51:24.199 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.197+0000 7f33d1ac2640 10 monclient: build_initial_monmap 2024-03-25T19:51:24.199 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.197+0000 7f33d1ac2640 10 monclient: monmap: 2024-03-25T19:51:24.199 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:51:24.199 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:24.199 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:51:24.200 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:51:24.200 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:24.200 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:51:24.200 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:51:24.200 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:51:24.200 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:51:24.200 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:24.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.197+0000 7f33d1ac2640 1 librados: starting msgr at 2024-03-25T19:51:24.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.197+0000 7f33d1ac2640 1 librados: starting objecter 2024-03-25T19:51:24.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.198+0000 7f33d1ac2640 1 Processor -- start 2024-03-25T19:51:24.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.198+0000 7f33d1ac2640 1 -- start start 2024-03-25T19:51:24.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.198+0000 7f33d1ac2640 1 librados: setting wanted keys 2024-03-25T19:51:24.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.198+0000 7f33d1ac2640 1 librados: calling monclient init 2024-03-25T19:51:24.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.198+0000 7f33d1ac2640 10 monclient: init 2024-03-25T19:51:24.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.198+0000 7f33d1ac2640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:24.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.198+0000 7f33d1ac2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:51:24.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.198+0000 7f33d1ac2640 10 monclient(hunting): picked mon.noname-a con 0x7f33cc0a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:51:24.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.198+0000 7f33d1ac2640 10 monclient(hunting): picked mon.noname-b con 0x7f33cc13f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:51:24.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.198+0000 7f33d1ac2640 10 monclient(hunting): picked mon.noname-c con 0x7f33cc1430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:51:24.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.198+0000 7f33d1ac2640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc138e90 con 0x7f33cc0a98f0 2024-03-25T19:51:24.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.198+0000 7f33d1ac2640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc0b1770 con 0x7f33cc13f710 2024-03-25T19:51:24.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.198+0000 7f33d1ac2640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc0abd30 con 0x7f33cc1430c0 2024-03-25T19:51:24.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.198+0000 7f33d1ac2640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:24.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.198+0000 7f33d1ac2640 10 monclient(hunting): authenticate will time out at 4519.280794s 2024-03-25T19:51:24.212 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.213+0000 7f33cbfff640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f33cc1430c0 0x7f33cc1e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:38428/0 (socket says 172.21.15.81:38428) 2024-03-25T19:51:24.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.213+0000 7f33cbfff640 1 -- 172.21.15.81:0/2774015471 learned_addr learned my addr 172.21.15.81:0/2774015471 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:51:24.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.214+0000 7f33b3fff640 10 client.?.objecter ms_handle_connect 0x7f33cc1430c0 2024-03-25T19:51:24.213 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.214+0000 7f33b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:24.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.224+0000 7f33b3fff640 10 client.?.objecter ms_handle_connect 0x7f33cc0a98f0 2024-03-25T19:51:24.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.224+0000 7f33b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:24.228 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.227+0000 7f33b3fff640 10 client.?.objecter ms_handle_connect 0x7f33cc13f710 2024-03-25T19:51:24.228 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.227+0000 7f33b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:24.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.237+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176100984 0 0) 0x7f33cc0abd30 con 0x7f33cc1430c0 2024-03-25T19:51:24.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.237+0000 7f33b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:24.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.237+0000 7f33b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:24.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.237+0000 7f33b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:24.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.237+0000 7f33b3fff640 10 monclient(hunting): my global_id is 4628 2024-03-25T19:51:24.237 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.237+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33b8004eb0 con 0x7f33cc1430c0 2024-03-25T19:51:24.245 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.246+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430046146 0 0) 0x7f33cc138e90 con 0x7f33cc0a98f0 2024-03-25T19:51:24.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.246+0000 7f33b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:24.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.246+0000 7f33b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:24.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.246+0000 7f33b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:24.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.246+0000 7f33b3fff640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:51:24.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.246+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33cc0abd30 con 0x7f33cc0a98f0 2024-03-25T19:51:24.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.246+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708769622 0 0) 0x7f33cc0b2a30 con 0x7f33cc13f710 2024-03-25T19:51:24.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.246+0000 7f33b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:24.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.246+0000 7f33b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:24.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.246+0000 7f33b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:24.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.246+0000 7f33b3fff640 10 monclient(hunting): my global_id is 4645 2024-03-25T19:51:24.247 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.246+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33cc138e90 con 0x7f33cc13f710 2024-03-25T19:51:24.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.258+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3950179632 0 0) 0x7f33cc138e90 con 0x7f33cc13f710 2024-03-25T19:51:24.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.259+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33cc0b2a30 con 0x7f33cc13f710 2024-03-25T19:51:24.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.259+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f33c0001690 con 0x7f33cc13f710 2024-03-25T19:51:24.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.259+0000 7f33b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:24.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.259+0000 7f33b3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:24.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.259+0000 7f33b3fff640 10 monclient(hunting): dump: 2024-03-25T19:51:24.259 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:24.259 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:24.259 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:24.259 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:24.259 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:24.259 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:24.259 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:24.259 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:24.259 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:24.259 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:24.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.259+0000 7f33b3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:24.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.263+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 75609225 0 0) 0x7f33cc0abd30 con 0x7f33cc0a98f0 2024-03-25T19:51:24.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.263+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33cc138e90 con 0x7f33cc0a98f0 2024-03-25T19:51:24.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.263+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f33bc001b70 con 0x7f33cc0a98f0 2024-03-25T19:51:24.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.263+0000 7f33b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:24.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.263+0000 7f33b3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:24.263 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.263+0000 7f33b3fff640 10 monclient(hunting): dump: 2024-03-25T19:51:24.263 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:24.263 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:24.263 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:24.263 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:24.263 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:24.264 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:24.264 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:24.264 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:24.264 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:24.264 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:24.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.271+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3745030620 0 0) 0x7f33b8004eb0 con 0x7f33cc1430c0 2024-03-25T19:51:24.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.271+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33cc0abd30 con 0x7f33cc1430c0 2024-03-25T19:51:24.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.271+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f33c4001ad0 con 0x7f33cc1430c0 2024-03-25T19:51:24.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.271+0000 7f33b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:24.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.271+0000 7f33b3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:24.271 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.271+0000 7f33b3fff640 10 monclient(hunting): dump: 2024-03-25T19:51:24.272 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:24.272 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:24.272 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:24.272 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:24.272 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:24.272 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:24.272 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:24.272 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:24.272 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:24.272 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:24.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2073742420 0 0) 0x7f33cc0b2a30 con 0x7f33cc13f710 2024-03-25T19:51:24.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 >> v1:172.21.15.81:6790/0 conn(0x7f33cc1430c0 legacy=0x7f33cc1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:24.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 >> v1:172.21.15.81:6789/0 conn(0x7f33cc0a98f0 legacy=0x7f33cc0ac4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:24.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33b3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:51:24.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33b3fff640 1 monclient: found mon.b 2024-03-25T19:51:24.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:24.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33b3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:51:24.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc0ac010 con 0x7f33cc13f710 2024-03-25T19:51:24.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33b3fff640 10 monclient: _finish_auth 0 2024-03-25T19:51:24.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33b3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:51:24.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:24.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33d1ac2640 5 monclient: authenticate success, global_id 4645 2024-03-25T19:51:24.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33d1ac2640 10 monclient: _renew_subs 2024-03-25T19:51:24.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33d1ac2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:51:24.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33d1ac2640 1 -- 172.21.15.81:0/2774015471 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33cc137cc0 con 0x7f33cc13f710 2024-03-25T19:51:24.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33d1ac2640 10 client.4645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:24.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33d1ac2640 10 monclient: _renew_subs 2024-03-25T19:51:24.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33d1ac2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:51:24.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33d1ac2640 1 -- 172.21.15.81:0/2774015471 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33cc136680 con 0x7f33cc13f710 2024-03-25T19:51:24.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.280+0000 7f33d1ac2640 1 librados: init done 2024-03-25T19:51:24.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.281+0000 7f33d1ac2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:24.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.281+0000 7f33d1ac2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:24.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.281+0000 7f33d1ac2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:51:24.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.281+0000 7f33d1ac2640 1 -- 172.21.15.81:0/2774015471 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33cc139210 con 0x7f33cc13f710 2024-03-25T19:51:24.577 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.577+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33c0001ba0 con 0x7f33cc13f710 2024-03-25T19:51:24.577 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.577+0000 7f33b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:24.577 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.577+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f33c0003790 con 0x7f33cc13f710 2024-03-25T19:51:24.577 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.577+0000 7f33b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:24.577 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.577+0000 7f33b3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:24.577 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.577+0000 7f33b3fff640 10 monclient: dump: 2024-03-25T19:51:24.577 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:24.577 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:24.577 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:24.577 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:24.577 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:24.578 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:24.578 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:24.578 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:24.578 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:24.578 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:24.578 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.577+0000 7f33c9ffb640 4 set_mon_vals no callback set 2024-03-25T19:51:24.578 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.578+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f33c0003970 con 0x7f33cc13f710 2024-03-25T19:51:24.579 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.579+0000 7f33cb7fe640 10 monclient: get_auth_request con 0x7f33b806e0b0 auth_method 0 2024-03-25T19:51:24.579 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.579+0000 7f33b3fff640 10 client.4645.objecter ms_handle_connect 0x7f33b806e0b0 2024-03-25T19:51:24.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.583+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f33c0001d10 con 0x7f33cc13f710 2024-03-25T19:51:24.582 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.583+0000 7f33b3fff640 10 client.4645.objecter ms_dispatch 0x7f33cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:51:24.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.583+0000 7f33b3fff640 3 client.4645.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:51:24.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.583+0000 7f33b3fff640 3 client.4645.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:51:24.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.583+0000 7f33b3fff640 20 client.4645.objecter dump_active .. 0 homeless 2024-03-25T19:51:24.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.583+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33c004ed60 con 0x7f33cc13f710 2024-03-25T19:51:24.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.583+0000 7f33b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:24.583 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.583+0000 7f33b3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:24.814 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.815+0000 7f33d1ac2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:24.815 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.815+0000 7f33d1ac2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:24.815 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.815+0000 7f33d1ac2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:51:24.815 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.815+0000 7f33d1ac2640 1 -- 172.21.15.81:0/2774015471 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33cc0b3ef0 con 0x7f33cc13f710 2024-03-25T19:51:24.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.863+0000 7f33b3fff640 1 -- 172.21.15.81:0/2774015471 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f33c0050e60 con 0x7f33cc13f710 2024-03-25T19:51:24.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.864+0000 7f33b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:24.863 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.864+0000 7f33b3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:24.863 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:51:24.864 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:24.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.866+0000 7f33d1ac2640 10 librados: watch_flush enter 2024-03-25T19:51:24.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.866+0000 7f33d1ac2640 10 librados: watch_flush exit 2024-03-25T19:51:24.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.866+0000 7f33d1ac2640 20 client.4645.objecter shutdown clearing up homeless session... 2024-03-25T19:51:24.866 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.866+0000 7f33d1ac2640 10 client.4645.objecter successfully canceled tick 2024-03-25T19:51:24.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.866+0000 7f33d1ac2640 1 -- 172.21.15.81:0/2774015471 >> v1:172.21.15.98:6800/3980829775 conn(0x7f33b806e0b0 legacy=0x7f33b8070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:24.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.866+0000 7f33d1ac2640 10 monclient: shutdown 2024-03-25T19:51:24.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.866+0000 7f33d1ac2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:24.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.866+0000 7f33d1ac2640 1 -- 172.21.15.81:0/2774015471 >> v1:172.21.15.98:6789/0 conn(0x7f33cc13f710 legacy=0x7f33cc138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:24.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.866+0000 7f33d1ac2640 1 -- 172.21.15.81:0/2774015471 shutdown_connections 2024-03-25T19:51:24.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.866+0000 7f33d1ac2640 1 -- 172.21.15.81:0/2774015471 >> 172.21.15.81:0/2774015471 conn(0x7f33cc0b7ea0 msgr2=0x7f33cc13d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:24.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.866+0000 7f33d1ac2640 1 -- 172.21.15.81:0/2774015471 shutdown_connections 2024-03-25T19:51:24.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.866+0000 7f33d1ac2640 1 -- 172.21.15.81:0/2774015471 wait complete. 2024-03-25T19:51:24.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:24.866+0000 7f33d1ac2640 1 librados: shutdown 2024-03-25T19:51:24.879 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:31.685 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:31.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.803+0000 7f1ec4038640 10 monclient: get_monmap_and_config 2024-03-25T19:51:31.803 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.803+0000 7f1ec4038640 10 monclient: build_initial_monmap 2024-03-25T19:51:31.803 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.803+0000 7f1ec4038640 10 monclient: monmap: 2024-03-25T19:51:31.803 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:51:31.803 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:31.803 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:51:31.804157+0000 2024-03-25T19:51:31.803 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:51:31.804157+0000 2024-03-25T19:51:31.803 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:31.803 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:31.803 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:51:31.803 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:51:31.803 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:51:31.803 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:31.804 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.803+0000 7f1ec4038640 1 Processor -- start 2024-03-25T19:51:31.804 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.804+0000 7f1ec4038640 1 -- start start 2024-03-25T19:51:31.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.804+0000 7f1ec4038640 10 monclient: init 2024-03-25T19:51:31.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.804+0000 7f1ec4038640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:31.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.804+0000 7f1ec4038640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:31.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.804+0000 7f1ec4038640 10 monclient(hunting): picked mon.noname-c con 0x7f1ebc0b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:51:31.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.804+0000 7f1ec4038640 10 monclient(hunting): picked mon.noname-a con 0x7f1ebc0b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:51:31.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.804+0000 7f1ec4038640 10 monclient(hunting): picked mon.noname-b con 0x7f1ebc14be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:51:31.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.804+0000 7f1ec4038640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc0af940 con 0x7f1ebc0b1730 2024-03-25T19:51:31.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.804+0000 7f1ec4038640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc0b0a70 con 0x7f1ebc0b4d60 2024-03-25T19:51:31.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.804+0000 7f1ec4038640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc0b0d50 con 0x7f1ebc14be90 2024-03-25T19:51:31.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.804+0000 7f1ec4038640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:31.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.804+0000 7f1ec4038640 10 monclient(hunting): authenticate will time out at 4526.886777s 2024-03-25T19:51:31.814 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.815+0000 7f1ec15d3640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f1ebc0b1730 0x7f1ebc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:49342/0 (socket says 172.21.15.81:49342) 2024-03-25T19:51:31.814 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.815+0000 7f1ec15d3640 1 -- 172.21.15.81:0/55388506 learned_addr learned my addr 172.21.15.81:0/55388506 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:51:31.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.831+0000 7f1ec0dd2640 1 -- 172.21.15.81:0/55388506 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695909229 0 0) 0x7f1ebc0af940 con 0x7f1ebc0b1730 2024-03-25T19:51:31.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.831+0000 7f1ec0dd2640 10 monclient(hunting): _negotiate 2024-03-25T19:51:31.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.831+0000 7f1ec0dd2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:31.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.831+0000 7f1ec0dd2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:31.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.831+0000 7f1ec0dd2640 10 monclient(hunting): my global_id is 4665 2024-03-25T19:51:31.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.831+0000 7f1ec0dd2640 1 -- 172.21.15.81:0/55388506 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea8003520 con 0x7f1ebc0b1730 2024-03-25T19:51:31.834 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.834+0000 7f1ec0dd2640 1 -- 172.21.15.81:0/55388506 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258941536 0 0) 0x7f1ebc0b0d50 con 0x7f1ebc14be90 2024-03-25T19:51:31.834 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.834+0000 7f1ec0dd2640 10 monclient(hunting): _negotiate 2024-03-25T19:51:31.834 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.834+0000 7f1ec0dd2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:31.834 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.834+0000 7f1ec0dd2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:31.834 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.834+0000 7f1ec0dd2640 10 monclient(hunting): my global_id is 4631 2024-03-25T19:51:31.834 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.834+0000 7f1ec0dd2640 1 -- 172.21.15.81:0/55388506 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ebc0af940 con 0x7f1ebc14be90 2024-03-25T19:51:31.841 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.841+0000 7f1ec0dd2640 1 -- 172.21.15.81:0/55388506 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942857411 0 0) 0x7f1ebc0b0a70 con 0x7f1ebc0b4d60 2024-03-25T19:51:31.841 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.841+0000 7f1ec0dd2640 10 monclient(hunting): _negotiate 2024-03-25T19:51:31.841 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.841+0000 7f1ec0dd2640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:31.841 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.841+0000 7f1ec0dd2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:31.841 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.841+0000 7f1ec0dd2640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:51:31.841 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.841+0000 7f1ec0dd2640 1 -- 172.21.15.81:0/55388506 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ebc0b0d50 con 0x7f1ebc0b4d60 2024-03-25T19:51:31.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec0dd2640 1 -- 172.21.15.81:0/55388506 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2391967450 0 0) 0x7f1ebc0af940 con 0x7f1ebc14be90 2024-03-25T19:51:31.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec0dd2640 1 -- 172.21.15.81:0/55388506 >> v1:172.21.15.98:6789/0 conn(0x7f1ebc0b4d60 legacy=0x7f1ebc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:31.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec0dd2640 1 -- 172.21.15.81:0/55388506 >> v1:172.21.15.81:6789/0 conn(0x7f1ebc0b1730 legacy=0x7f1ebc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:31.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec0dd2640 10 monclient: _finish_hunting 0 2024-03-25T19:51:31.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec0dd2640 1 monclient: found mon.noname-b 2024-03-25T19:51:31.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec0dd2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:31.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec0dd2640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:51:31.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec0dd2640 1 -- 172.21.15.81:0/55388506 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc0b1030 con 0x7f1ebc14be90 2024-03-25T19:51:31.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec0dd2640 10 monclient: _finish_auth 0 2024-03-25T19:51:31.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec0dd2640 10 monclient: _check_auth_tickets 2024-03-25T19:51:31.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec0dd2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:31.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec0dd2640 1 -- 172.21.15.81:0/55388506 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1eb40019f0 con 0x7f1ebc14be90 2024-03-25T19:51:31.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec4038640 5 monclient: authenticate success, global_id 4631 2024-03-25T19:51:31.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec4038640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:31.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec0dd2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:31.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec0dd2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:31.847 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec0dd2640 10 monclient: dump: 2024-03-25T19:51:31.847 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:31.847 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:31.847 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:31.847 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:31.847 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:31.847 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:31.847 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:31.847 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:31.848 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:31.848 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:31.848 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.846+0000 7f1ec4038640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:31.855 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.856+0000 7f1ec0dd2640 1 -- 172.21.15.81:0/55388506 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1eb4001b50 con 0x7f1ebc14be90 2024-03-25T19:51:31.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.856+0000 7f1ec0dd2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:31.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.856+0000 7f1ec0dd2640 1 -- 172.21.15.81:0/55388506 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1eb4001f60 con 0x7f1ebc14be90 2024-03-25T19:51:31.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.856+0000 7f1ec4038640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:31.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.856+0000 7f1ec4038640 4 set_mon_vals no callback set 2024-03-25T19:51:31.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.856+0000 7f1ec0dd2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:31.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.856+0000 7f1ec0dd2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:31.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.856+0000 7f1ec0dd2640 10 monclient: dump: 2024-03-25T19:51:31.856 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:31.856 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:31.856 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:31.856 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:31.856 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:31.856 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:31.856 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:31.856 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:31.856 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:31.857 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:31.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.856+0000 7f1ec4038640 10 monclient: shutdown 2024-03-25T19:51:31.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.856+0000 7f1ec4038640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:31.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.856+0000 7f1ec4038640 1 -- 172.21.15.81:0/55388506 >> v1:172.21.15.81:6790/0 conn(0x7f1ebc14be90 legacy=0x7f1ebc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:31.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.856+0000 7f1ec4038640 1 -- 172.21.15.81:0/55388506 shutdown_connections 2024-03-25T19:51:31.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.857+0000 7f1ec4038640 1 -- 172.21.15.81:0/55388506 >> 172.21.15.81:0/55388506 conn(0x7f1ebc0617a0 msgr2=0x7f1ebc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:31.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.857+0000 7f1ec4038640 1 -- 172.21.15.81:0/55388506 shutdown_connections 2024-03-25T19:51:31.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.857+0000 7f1ec4038640 1 -- 172.21.15.81:0/55388506 wait complete. 2024-03-25T19:51:31.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.857+0000 7f1ec4038640 10 monclient: build_initial_monmap 2024-03-25T19:51:31.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.857+0000 7f1ec4038640 10 monclient: monmap: 2024-03-25T19:51:31.858 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:51:31.858 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:31.859 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:51:31.859 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:51:31.859 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:31.859 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:51:31.859 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:51:31.859 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:51:31.859 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:51:31.859 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:31.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.858+0000 7f1ec4038640 1 librados: starting msgr at 2024-03-25T19:51:31.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.858+0000 7f1ec4038640 1 librados: starting objecter 2024-03-25T19:51:31.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.858+0000 7f1ec4038640 1 Processor -- start 2024-03-25T19:51:31.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.858+0000 7f1ec4038640 1 -- start start 2024-03-25T19:51:31.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.858+0000 7f1ec4038640 1 librados: setting wanted keys 2024-03-25T19:51:31.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.858+0000 7f1ec4038640 1 librados: calling monclient init 2024-03-25T19:51:31.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.858+0000 7f1ec4038640 10 monclient: init 2024-03-25T19:51:31.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.859+0000 7f1ec4038640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:31.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.859+0000 7f1ec4038640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:31.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.859+0000 7f1ec4038640 10 monclient(hunting): picked mon.noname-a con 0x7f1ebc0b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:51:31.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.859+0000 7f1ec4038640 10 monclient(hunting): picked mon.noname-c con 0x7f1ebc0b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:51:31.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.859+0000 7f1ec4038640 10 monclient(hunting): picked mon.noname-b con 0x7f1ebc14be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:51:31.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.859+0000 7f1ec4038640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc1419c0 con 0x7f1ebc0b1730 2024-03-25T19:51:31.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.859+0000 7f1ec4038640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc13bbc0 con 0x7f1ebc14be90 2024-03-25T19:51:31.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.859+0000 7f1ec4038640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ebc13bd50 con 0x7f1ebc0b4d60 2024-03-25T19:51:31.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.859+0000 7f1ec4038640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:31.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.859+0000 7f1ec4038640 10 monclient(hunting): authenticate will time out at 4526.941301s 2024-03-25T19:51:31.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.861+0000 7f1ec15d3640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f1ebc0b4d60 0x7f1ebc1d7ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:60954/0 (socket says 172.21.15.81:60954) 2024-03-25T19:51:31.861 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.861+0000 7f1ec15d3640 1 -- 172.21.15.81:0/309248220 learned_addr learned my addr 172.21.15.81:0/309248220 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:51:31.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.865+0000 7f1eb27fc640 10 client.?.objecter ms_handle_connect 0x7f1ebc0b1730 2024-03-25T19:51:31.865 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.865+0000 7f1eb27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:31.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.868+0000 7f1eb27fc640 10 client.?.objecter ms_handle_connect 0x7f1ebc0b4d60 2024-03-25T19:51:31.867 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.868+0000 7f1eb27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:31.873 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.874+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424551370 0 0) 0x7f1ebc1419c0 con 0x7f1ebc0b1730 2024-03-25T19:51:31.873 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.874+0000 7f1eb27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:31.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.874+0000 7f1eb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:31.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.874+0000 7f1eb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:31.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.874+0000 7f1eb27fc640 10 monclient(hunting): my global_id is 4668 2024-03-25T19:51:31.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.874+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea0004eb0 con 0x7f1ebc0b1730 2024-03-25T19:51:31.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.874+0000 7f1eb27fc640 10 client.?.objecter ms_handle_connect 0x7f1ebc14be90 2024-03-25T19:51:31.874 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.874+0000 7f1eb27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:31.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.886+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395512059 0 0) 0x7f1ebc13bbc0 con 0x7f1ebc14be90 2024-03-25T19:51:31.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.886+0000 7f1eb27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:31.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.886+0000 7f1eb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:31.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.886+0000 7f1eb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:31.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.886+0000 7f1eb27fc640 10 monclient(hunting): my global_id is 4651 2024-03-25T19:51:31.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.886+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ebc1419c0 con 0x7f1ebc14be90 2024-03-25T19:51:31.892 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.892+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202861540 0 0) 0x7f1ebc13bd50 con 0x7f1ebc0b4d60 2024-03-25T19:51:31.892 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.892+0000 7f1eb27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:31.892 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.892+0000 7f1eb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:31.892 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.892+0000 7f1eb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:31.892 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.892+0000 7f1eb27fc640 10 monclient(hunting): my global_id is 4634 2024-03-25T19:51:31.892 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.892+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ebc13bbc0 con 0x7f1ebc0b4d60 2024-03-25T19:51:31.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.902+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3056007354 0 0) 0x7f1ea0004eb0 con 0x7f1ebc0b1730 2024-03-25T19:51:31.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.902+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ebc13bd50 con 0x7f1ebc0b1730 2024-03-25T19:51:31.902 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.902+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1eb8001970 con 0x7f1ebc0b1730 2024-03-25T19:51:31.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.902+0000 7f1eb27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:31.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.902+0000 7f1eb27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:31.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.902+0000 7f1eb27fc640 10 monclient(hunting): dump: 2024-03-25T19:51:31.903 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:31.903 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:31.903 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:31.903 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:31.903 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:31.903 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:31.903 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:31.903 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:31.903 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:31.903 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:31.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.902+0000 7f1eb27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:31.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.902+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1821951169 0 0) 0x7f1ebc1419c0 con 0x7f1ebc14be90 2024-03-25T19:51:31.903 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.902+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ea0004eb0 con 0x7f1ebc14be90 2024-03-25T19:51:31.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.902+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1eb4001e60 con 0x7f1ebc14be90 2024-03-25T19:51:31.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.902+0000 7f1eb27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:31.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.902+0000 7f1eb27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:31.904 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.902+0000 7f1eb27fc640 10 monclient(hunting): dump: 2024-03-25T19:51:31.904 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:31.904 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:31.904 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:31.904 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:31.904 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:31.904 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:31.904 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:31.904 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:31.904 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:31.904 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:31.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.904+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1416297396 0 0) 0x7f1ebc13bbc0 con 0x7f1ebc0b4d60 2024-03-25T19:51:31.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.904+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ebc1419c0 con 0x7f1ebc0b4d60 2024-03-25T19:51:31.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.904+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1eac001950 con 0x7f1ebc0b4d60 2024-03-25T19:51:31.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.904+0000 7f1eb27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:31.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.904+0000 7f1eb27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:31.905 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.904+0000 7f1eb27fc640 10 monclient(hunting): dump: 2024-03-25T19:51:31.905 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:31.905 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:31.905 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:31.905 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:31.906 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:31.906 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:31.906 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:31.906 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:31.906 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:31.906 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:31.908 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 295652311 0 0) 0x7f1ebc13bd50 con 0x7f1ebc0b1730 2024-03-25T19:51:31.908 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 >> v1:172.21.15.81:6790/0 conn(0x7f1ebc0b4d60 legacy=0x7f1ebc1d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:31.908 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 >> v1:172.21.15.98:6789/0 conn(0x7f1ebc14be90 legacy=0x7f1ebc1e20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:31.908 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1eb27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:51:31.908 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1eb27fc640 1 monclient: found mon.a 2024-03-25T19:51:31.908 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1eb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:31.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1eb27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:51:31.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc13b060 con 0x7f1ebc0b1730 2024-03-25T19:51:31.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1eb27fc640 10 monclient: _finish_auth 0 2024-03-25T19:51:31.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1eb27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:51:31.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1eb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:31.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1ec4038640 5 monclient: authenticate success, global_id 4668 2024-03-25T19:51:31.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1ec4038640 10 monclient: _renew_subs 2024-03-25T19:51:31.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1ec4038640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:51:31.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1ec4038640 1 -- 172.21.15.81:0/309248220 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ebc1421a0 con 0x7f1ebc0b1730 2024-03-25T19:51:31.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1ec4038640 10 client.4668.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:31.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1ec4038640 10 monclient: _renew_subs 2024-03-25T19:51:31.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1ec4038640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:51:31.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1ec4038640 1 -- 172.21.15.81:0/309248220 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ebc1d8590 con 0x7f1ebc0b1730 2024-03-25T19:51:31.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.908+0000 7f1ec4038640 1 librados: init done 2024-03-25T19:51:31.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.909+0000 7f1ec4038640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:31.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.909+0000 7f1ec4038640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:31.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.909+0000 7f1ec4038640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:51:31.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:31.909+0000 7f1ec4038640 1 -- 172.21.15.81:0/309248220 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ebc13b250 con 0x7f1ebc0b1730 2024-03-25T19:51:32.136 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.137+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1eb8003610 con 0x7f1ebc0b1730 2024-03-25T19:51:32.137 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.137+0000 7f1eb27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:32.137 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.137+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1eb8003920 con 0x7f1ebc0b1730 2024-03-25T19:51:32.137 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.137+0000 7f1eb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:32.137 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.137+0000 7f1eb27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:32.137 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.137+0000 7f1eb27fc640 10 monclient: dump: 2024-03-25T19:51:32.137 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:32.137 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:32.137 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:32.137 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:32.137 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:32.137 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:32.137 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:32.137 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:32.137 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:32.138 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:32.138 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.137+0000 7f1eb3fff640 4 set_mon_vals no callback set 2024-03-25T19:51:32.218 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.219+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f1eb8019970 con 0x7f1ebc0b1730 2024-03-25T19:51:32.219 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.219+0000 7f1ec15d3640 10 monclient: get_auth_request con 0x7f1ea006e0b0 auth_method 0 2024-03-25T19:51:32.219 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.220+0000 7f1eb27fc640 10 client.4668.objecter ms_handle_connect 0x7f1ea006e0b0 2024-03-25T19:51:32.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.227+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f1eb8053670 con 0x7f1ebc0b1730 2024-03-25T19:51:32.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.227+0000 7f1eb27fc640 10 client.4668.objecter ms_dispatch 0x7f1ebc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:51:32.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.227+0000 7f1eb27fc640 3 client.4668.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:51:32.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.227+0000 7f1eb27fc640 3 client.4668.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:51:32.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.227+0000 7f1eb27fc640 20 client.4668.objecter dump_active .. 0 homeless 2024-03-25T19:51:32.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.227+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1eb804efb0 con 0x7f1ebc0b1730 2024-03-25T19:51:32.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.227+0000 7f1eb27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:32.228 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.227+0000 7f1eb27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:32.470 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.470+0000 7f1ec4038640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:32.470 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.470+0000 7f1ec4038640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:32.470 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.470+0000 7f1ec4038640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:51:32.470 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.470+0000 7f1ec4038640 1 -- 172.21.15.81:0/309248220 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ebc0627b0 con 0x7f1ebc0b1730 2024-03-25T19:51:32.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.512+0000 7f1eb27fc640 1 -- 172.21.15.81:0/309248220 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f1eb80510b0 con 0x7f1ebc0b1730 2024-03-25T19:51:32.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.513+0000 7f1eb27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:32.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.513+0000 7f1eb27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:32.513 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:51:32.513 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:32.515 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.515+0000 7f1ec4038640 10 librados: watch_flush enter 2024-03-25T19:51:32.515 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.515+0000 7f1ec4038640 10 librados: watch_flush exit 2024-03-25T19:51:32.516 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.515+0000 7f1ec4038640 20 client.4668.objecter shutdown clearing up homeless session... 2024-03-25T19:51:32.516 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.515+0000 7f1ec4038640 10 client.4668.objecter successfully canceled tick 2024-03-25T19:51:32.516 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.515+0000 7f1ec4038640 1 -- 172.21.15.81:0/309248220 >> v1:172.21.15.98:6800/3980829775 conn(0x7f1ea006e0b0 legacy=0x7f1ea0070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:32.516 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.515+0000 7f1ec4038640 10 monclient: shutdown 2024-03-25T19:51:32.516 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.515+0000 7f1ec4038640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:32.516 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.515+0000 7f1ec4038640 1 -- 172.21.15.81:0/309248220 >> v1:172.21.15.81:6789/0 conn(0x7f1ebc0b1730 legacy=0x7f1ebc1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:32.516 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.515+0000 7f1ec4038640 1 -- 172.21.15.81:0/309248220 shutdown_connections 2024-03-25T19:51:32.516 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.515+0000 7f1ec4038640 1 -- 172.21.15.81:0/309248220 >> 172.21.15.81:0/309248220 conn(0x7f1ebc0617a0 msgr2=0x7f1ebc13c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:32.516 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.515+0000 7f1ec4038640 1 -- 172.21.15.81:0/309248220 shutdown_connections 2024-03-25T19:51:32.516 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.515+0000 7f1ec4038640 1 -- 172.21.15.81:0/309248220 wait complete. 2024-03-25T19:51:32.516 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:32.515+0000 7f1ec4038640 1 librados: shutdown 2024-03-25T19:51:32.528 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:39.335 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:39.453 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.453+0000 7f01fd48d640 10 monclient: get_monmap_and_config 2024-03-25T19:51:39.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.453+0000 7f01fd48d640 10 monclient: build_initial_monmap 2024-03-25T19:51:39.454 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.454+0000 7f01fd48d640 10 monclient: monmap: 2024-03-25T19:51:39.454 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:51:39.454 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:39.454 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:51:39.454879+0000 2024-03-25T19:51:39.454 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:51:39.454879+0000 2024-03-25T19:51:39.454 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:39.454 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:39.454 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:51:39.454 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:51:39.455 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:51:39.455 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:39.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.454+0000 7f01fd48d640 1 Processor -- start 2024-03-25T19:51:39.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.454+0000 7f01fd48d640 1 -- start start 2024-03-25T19:51:39.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.454+0000 7f01fd48d640 10 monclient: init 2024-03-25T19:51:39.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.455+0000 7f01fd48d640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:39.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.455+0000 7f01fd48d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:51:39.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.455+0000 7f01fd48d640 10 monclient(hunting): picked mon.noname-b con 0x7f01f80a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:51:39.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.455+0000 7f01fd48d640 10 monclient(hunting): picked mon.noname-c con 0x7f01f813f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:51:39.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.455+0000 7f01fd48d640 10 monclient(hunting): picked mon.noname-a con 0x7f01f81430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:51:39.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.455+0000 7f01fd48d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f8142520 con 0x7f01f81430c0 2024-03-25T19:51:39.456 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.455+0000 7f01fd48d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f80b28a0 con 0x7f01f813f710 2024-03-25T19:51:39.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.455+0000 7f01fd48d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f80b2a30 con 0x7f01f80a98f0 2024-03-25T19:51:39.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.455+0000 7f01fd48d640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:39.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.455+0000 7f01fd48d640 10 monclient(hunting): authenticate will time out at 4534.537311s 2024-03-25T19:51:39.462 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.463+0000 7f01f6ffd640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f01f80a98f0 0x7f01f80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:33178/0 (socket says 172.21.15.81:33178) 2024-03-25T19:51:39.462 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.463+0000 7f01f6ffd640 1 -- 172.21.15.81:0/799875177 learned_addr learned my addr 172.21.15.81:0/799875177 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:51:39.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.476+0000 7f01f5ffb640 1 -- 172.21.15.81:0/799875177 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944330103 0 0) 0x7f01f80b28a0 con 0x7f01f813f710 2024-03-25T19:51:39.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.476+0000 7f01f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:39.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.476+0000 7f01f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:39.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.476+0000 7f01f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:39.476 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.476+0000 7f01f5ffb640 10 monclient(hunting): my global_id is 4654 2024-03-25T19:51:39.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.476+0000 7f01f5ffb640 1 -- 172.21.15.81:0/799875177 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01e0003520 con 0x7f01f813f710 2024-03-25T19:51:39.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.479+0000 7f01f5ffb640 1 -- 172.21.15.81:0/799875177 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097055223 0 0) 0x7f01f80b2a30 con 0x7f01f80a98f0 2024-03-25T19:51:39.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.479+0000 7f01f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:39.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.479+0000 7f01f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:39.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.479+0000 7f01f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:39.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.479+0000 7f01f5ffb640 10 monclient(hunting): my global_id is 4637 2024-03-25T19:51:39.480 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.479+0000 7f01f5ffb640 1 -- 172.21.15.81:0/799875177 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f80b28a0 con 0x7f01f80a98f0 2024-03-25T19:51:39.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.485+0000 7f01f5ffb640 1 -- 172.21.15.81:0/799875177 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2341299658 0 0) 0x7f01e0003520 con 0x7f01f813f710 2024-03-25T19:51:39.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.485+0000 7f01f5ffb640 1 -- 172.21.15.81:0/799875177 >> v1:172.21.15.81:6790/0 conn(0x7f01f80a98f0 legacy=0x7f01f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:39.485 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.485+0000 7f01f5ffb640 1 -- 172.21.15.81:0/799875177 >> v1:172.21.15.81:6789/0 conn(0x7f01f81430c0 legacy=0x7f01f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:39.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.485+0000 7f01f5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:51:39.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.485+0000 7f01f5ffb640 1 monclient: found mon.noname-c 2024-03-25T19:51:39.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.485+0000 7f01f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:39.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.485+0000 7f01f5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:51:39.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.485+0000 7f01f5ffb640 1 -- 172.21.15.81:0/799875177 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f80b44c0 con 0x7f01f813f710 2024-03-25T19:51:39.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.485+0000 7f01f5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:51:39.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.485+0000 7f01f5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:51:39.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.486+0000 7f01f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:39.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.486+0000 7f01f5ffb640 1 -- 172.21.15.81:0/799875177 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f01ec001690 con 0x7f01f813f710 2024-03-25T19:51:39.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.486+0000 7f01f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:39.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.486+0000 7f01f5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:39.486 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.486+0000 7f01f5ffb640 10 monclient: dump: 2024-03-25T19:51:39.486 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:39.487 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:39.487 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:39.487 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:39.487 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:39.487 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:39.487 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:39.487 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:39.487 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:39.487 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:39.487 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.486+0000 7f01fd48d640 5 monclient: authenticate success, global_id 4654 2024-03-25T19:51:39.487 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.486+0000 7f01fd48d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:39.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.488+0000 7f01f5ffb640 1 -- 172.21.15.81:0/799875177 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01ec0017f0 con 0x7f01f813f710 2024-03-25T19:51:39.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.488+0000 7f01f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:39.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.488+0000 7f01fd48d640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:39.488 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.488+0000 7f01fd48d640 4 set_mon_vals no callback set 2024-03-25T19:51:39.489 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.489+0000 7f01fd48d640 10 monclient: shutdown 2024-03-25T19:51:39.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.489+0000 7f01fd48d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:39.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.489+0000 7f01fd48d640 1 -- 172.21.15.81:0/799875177 >> v1:172.21.15.98:6789/0 conn(0x7f01f813f710 legacy=0x7f01f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:39.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.489+0000 7f01fd48d640 1 -- 172.21.15.81:0/799875177 shutdown_connections 2024-03-25T19:51:39.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.489+0000 7f01fd48d640 1 -- 172.21.15.81:0/799875177 >> 172.21.15.81:0/799875177 conn(0x7f01f80b7ea0 msgr2=0x7f01f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:39.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.489+0000 7f01fd48d640 1 -- 172.21.15.81:0/799875177 shutdown_connections 2024-03-25T19:51:39.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.489+0000 7f01fd48d640 1 -- 172.21.15.81:0/799875177 wait complete. 2024-03-25T19:51:39.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.490+0000 7f01fd48d640 10 monclient: build_initial_monmap 2024-03-25T19:51:39.490 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.490+0000 7f01fd48d640 10 monclient: monmap: 2024-03-25T19:51:39.490 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:51:39.490 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:39.491 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:51:39.491 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:51:39.491 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:39.491 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:51:39.491 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:51:39.491 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:51:39.491 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:51:39.491 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:39.491 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.490+0000 7f01fd48d640 1 librados: starting msgr at 2024-03-25T19:51:39.491 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.490+0000 7f01fd48d640 1 librados: starting objecter 2024-03-25T19:51:39.492 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.490+0000 7f01fd48d640 1 Processor -- start 2024-03-25T19:51:39.492 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.490+0000 7f01fd48d640 1 -- start start 2024-03-25T19:51:39.492 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.491+0000 7f01fd48d640 1 librados: setting wanted keys 2024-03-25T19:51:39.492 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.491+0000 7f01fd48d640 1 librados: calling monclient init 2024-03-25T19:51:39.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.491+0000 7f01fd48d640 10 monclient: init 2024-03-25T19:51:39.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.491+0000 7f01fd48d640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:39.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.491+0000 7f01fd48d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:51:39.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.491+0000 7f01fd48d640 10 monclient(hunting): picked mon.noname-b con 0x7f01f80a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:51:39.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.491+0000 7f01fd48d640 10 monclient(hunting): picked mon.noname-c con 0x7f01f813f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:51:39.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.491+0000 7f01fd48d640 10 monclient(hunting): picked mon.noname-a con 0x7f01f81430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:51:39.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.491+0000 7f01fd48d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f8136a10 con 0x7f01f81430c0 2024-03-25T19:51:39.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.491+0000 7f01fd48d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f8133b60 con 0x7f01f80a98f0 2024-03-25T19:51:39.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.491+0000 7f01fd48d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f80ab950 con 0x7f01f813f710 2024-03-25T19:51:39.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.491+0000 7f01fd48d640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:39.493 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.491+0000 7f01fd48d640 10 monclient(hunting): authenticate will time out at 4534.573508s 2024-03-25T19:51:39.495 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.495+0000 7f01f67fc640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f01f813f710 0x7f01f8136300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:33208/0 (socket says 172.21.15.81:33208) 2024-03-25T19:51:39.495 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.495+0000 7f01f67fc640 1 -- 172.21.15.81:0/3314978123 learned_addr learned my addr 172.21.15.81:0/3314978123 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:51:39.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.496+0000 7f01df7fe640 10 client.?.objecter ms_handle_connect 0x7f01f813f710 2024-03-25T19:51:39.496 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.496+0000 7f01df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:39.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.498+0000 7f01df7fe640 10 client.?.objecter ms_handle_connect 0x7f01f81430c0 2024-03-25T19:51:39.497 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.498+0000 7f01df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:39.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.504+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263136191 0 0) 0x7f01f80ab950 con 0x7f01f813f710 2024-03-25T19:51:39.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.504+0000 7f01df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:39.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.504+0000 7f01df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:39.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.504+0000 7f01df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:39.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.504+0000 7f01df7fe640 10 monclient(hunting): my global_id is 4640 2024-03-25T19:51:39.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.504+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01e4004eb0 con 0x7f01f813f710 2024-03-25T19:51:39.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.505+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653378393 0 0) 0x7f01f8136a10 con 0x7f01f81430c0 2024-03-25T19:51:39.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.505+0000 7f01df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:39.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.505+0000 7f01df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:39.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.505+0000 7f01df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:39.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.505+0000 7f01df7fe640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:51:39.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.505+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f80ab950 con 0x7f01f81430c0 2024-03-25T19:51:39.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.513+0000 7f01df7fe640 10 client.?.objecter ms_handle_connect 0x7f01f80a98f0 2024-03-25T19:51:39.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.513+0000 7f01df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:39.520 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.520+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2398761195 0 0) 0x7f01e4004eb0 con 0x7f01f813f710 2024-03-25T19:51:39.520 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.520+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01f8136a10 con 0x7f01f813f710 2024-03-25T19:51:39.520 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.520+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f01ec001d60 con 0x7f01f813f710 2024-03-25T19:51:39.520 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.520+0000 7f01df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:39.520 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.520+0000 7f01df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:39.520 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.520+0000 7f01df7fe640 10 monclient(hunting): dump: 2024-03-25T19:51:39.521 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:39.521 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:39.521 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:39.521 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:39.521 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:39.521 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:39.521 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:39.521 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:39.521 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:39.522 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:39.522 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.520+0000 7f01df7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:39.532 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.532+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 404356526 0 0) 0x7f01f80ab950 con 0x7f01f81430c0 2024-03-25T19:51:39.532 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.532+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01e4004eb0 con 0x7f01f81430c0 2024-03-25T19:51:39.532 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.532+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f01f0001b50 con 0x7f01f81430c0 2024-03-25T19:51:39.532 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.532+0000 7f01df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:39.532 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.532+0000 7f01df7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:39.532 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.532+0000 7f01df7fe640 10 monclient(hunting): dump: 2024-03-25T19:51:39.532 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:39.532 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:39.532 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:39.533 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:39.533 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:39.533 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:39.533 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:39.533 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:39.533 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:39.533 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:39.533 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.532+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388217758 0 0) 0x7f01f8133b60 con 0x7f01f80a98f0 2024-03-25T19:51:39.533 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.532+0000 7f01df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:39.533 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.532+0000 7f01df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:39.533 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.532+0000 7f01df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:39.533 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.532+0000 7f01df7fe640 10 monclient(hunting): my global_id is 4657 2024-03-25T19:51:39.533 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.532+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f80ab950 con 0x7f01f80a98f0 2024-03-25T19:51:39.537 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 313034328 0 0) 0x7f01f8136a10 con 0x7f01f813f710 2024-03-25T19:51:39.537 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 >> v1:172.21.15.98:6789/0 conn(0x7f01f80a98f0 legacy=0x7f01f8132d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:39.537 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 >> v1:172.21.15.81:6789/0 conn(0x7f01f81430c0 legacy=0x7f01f8133450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:39.537 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01df7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:39.537 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01df7fe640 1 monclient: found mon.c 2024-03-25T19:51:39.537 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:39.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01df7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:51:39.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f80abc30 con 0x7f01f813f710 2024-03-25T19:51:39.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01df7fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:39.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01df7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:39.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:39.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01fd48d640 5 monclient: authenticate success, global_id 4640 2024-03-25T19:51:39.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01fd48d640 10 monclient: _renew_subs 2024-03-25T19:51:39.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01fd48d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:51:39.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01fd48d640 1 -- 172.21.15.81:0/3314978123 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f80abf10 con 0x7f01f813f710 2024-03-25T19:51:39.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01fd48d640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:39.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01fd48d640 10 monclient: _renew_subs 2024-03-25T19:51:39.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01fd48d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:51:39.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01fd48d640 1 -- 172.21.15.81:0/3314978123 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f8137e00 con 0x7f01f813f710 2024-03-25T19:51:39.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.537+0000 7f01fd48d640 1 librados: init done 2024-03-25T19:51:39.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.538+0000 7f01fd48d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:39.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.538+0000 7f01fd48d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:39.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.538+0000 7f01fd48d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:51:39.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.538+0000 7f01fd48d640 1 -- 172.21.15.81:0/3314978123 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f80627b0 con 0x7f01f813f710 2024-03-25T19:51:39.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.830+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01ec0038a0 con 0x7f01f813f710 2024-03-25T19:51:39.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.830+0000 7f01df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:39.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.830+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f01ec001250 con 0x7f01f813f710 2024-03-25T19:51:39.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.830+0000 7f01df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:39.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.830+0000 7f01df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:39.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.830+0000 7f01df7fe640 10 monclient: dump: 2024-03-25T19:51:39.830 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:39.830 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:39.831 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:39.831 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:39.831 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:39.831 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:39.831 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:39.831 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:39.831 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:39.831 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:39.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.830+0000 7f01f57fa640 4 set_mon_vals no callback set 2024-03-25T19:51:39.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.831+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f01ec003d70 con 0x7f01f813f710 2024-03-25T19:51:39.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.831+0000 7f01f6ffd640 10 monclient: get_auth_request con 0x7f01e406df00 auth_method 0 2024-03-25T19:51:39.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.832+0000 7f01df7fe640 10 client.4640.objecter ms_handle_connect 0x7f01e406df00 2024-03-25T19:51:39.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.840+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f01ec054980 con 0x7f01f813f710 2024-03-25T19:51:39.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.840+0000 7f01df7fe640 10 client.4640.objecter ms_dispatch 0x7f01f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:51:39.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.840+0000 7f01df7fe640 3 client.4640.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:51:39.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.840+0000 7f01df7fe640 3 client.4640.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:51:39.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.840+0000 7f01df7fe640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-25T19:51:39.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.840+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f01ec050490 con 0x7f01f813f710 2024-03-25T19:51:39.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.840+0000 7f01df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:39.840 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:39.840+0000 7f01df7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:40.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.052+0000 7f01fd48d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:40.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.052+0000 7f01fd48d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:40.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.052+0000 7f01fd48d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:51:40.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.052+0000 7f01fd48d640 1 -- 172.21.15.81:0/3314978123 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01f813c060 con 0x7f01f813f710 2024-03-25T19:51:40.085 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.086+0000 7f01df7fe640 1 -- 172.21.15.81:0/3314978123 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f01ec0523c0 con 0x7f01f813f710 2024-03-25T19:51:40.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.086+0000 7f01df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:40.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.086+0000 7f01df7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:40.086 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:51:40.086 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:40.088 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.088+0000 7f01fd48d640 10 librados: watch_flush enter 2024-03-25T19:51:40.088 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.088+0000 7f01fd48d640 10 librados: watch_flush exit 2024-03-25T19:51:40.088 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.088+0000 7f01fd48d640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-25T19:51:40.088 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.088+0000 7f01fd48d640 10 client.4640.objecter successfully canceled tick 2024-03-25T19:51:40.088 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.088+0000 7f01fd48d640 1 -- 172.21.15.81:0/3314978123 >> v1:172.21.15.98:6800/3980829775 conn(0x7f01e406df00 legacy=0x7f01e40703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:40.088 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.088+0000 7f01fd48d640 10 monclient: shutdown 2024-03-25T19:51:40.088 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.088+0000 7f01fd48d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:40.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.088+0000 7f01fd48d640 1 -- 172.21.15.81:0/3314978123 >> v1:172.21.15.81:6790/0 conn(0x7f01f813f710 legacy=0x7f01f8136300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:40.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.088+0000 7f01fd48d640 1 -- 172.21.15.81:0/3314978123 shutdown_connections 2024-03-25T19:51:40.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.088+0000 7f01fd48d640 1 -- 172.21.15.81:0/3314978123 >> 172.21.15.81:0/3314978123 conn(0x7f01f80b7ea0 msgr2=0x7f01f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:40.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.088+0000 7f01fd48d640 1 -- 172.21.15.81:0/3314978123 shutdown_connections 2024-03-25T19:51:40.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.088+0000 7f01fd48d640 1 -- 172.21.15.81:0/3314978123 wait complete. 2024-03-25T19:51:40.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:40.088+0000 7f01fd48d640 1 librados: shutdown 2024-03-25T19:51:40.101 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:43.645 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:10:06.286 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:43.693 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:10:06.335 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:43.695 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:10:06.337 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:43.696 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:10:06.338 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:43.706 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:10:06.348 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:43.729 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:10:06.371 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:43.751 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:06.386 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:43.760 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:06.395 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:43.793 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:06.427 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:46.907 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:47.026 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.026+0000 7fc515c86640 10 monclient: get_monmap_and_config 2024-03-25T19:51:47.026 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.026+0000 7fc515c86640 10 monclient: build_initial_monmap 2024-03-25T19:51:47.027 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.026+0000 7fc515c86640 10 monclient: monmap: 2024-03-25T19:51:47.027 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:51:47.027 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:47.027 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:51:47.027798+0000 2024-03-25T19:51:47.027 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:51:47.027798+0000 2024-03-25T19:51:47.027 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:47.027 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:47.027 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:51:47.027 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:51:47.027 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:51:47.027 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:47.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.027+0000 7fc515c86640 1 Processor -- start 2024-03-25T19:51:47.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.027+0000 7fc515c86640 1 -- start start 2024-03-25T19:51:47.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.027+0000 7fc515c86640 10 monclient: init 2024-03-25T19:51:47.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.028+0000 7fc515c86640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:47.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.028+0000 7fc515c86640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:51:47.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.028+0000 7fc515c86640 10 monclient(hunting): picked mon.noname-c con 0x7fc5100a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:51:47.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.028+0000 7fc515c86640 10 monclient(hunting): picked mon.noname-b con 0x7fc51013f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:51:47.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.028+0000 7fc515c86640 10 monclient(hunting): picked mon.noname-a con 0x7fc5101430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:51:47.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.028+0000 7fc515c86640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc510142520 con 0x7fc5101430c0 2024-03-25T19:51:47.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.028+0000 7fc515c86640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5100b28a0 con 0x7fc5100a98f0 2024-03-25T19:51:47.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.028+0000 7fc515c86640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5100b2a30 con 0x7fc51013f710 2024-03-25T19:51:47.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.028+0000 7fc515c86640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:47.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.028+0000 7fc515c86640 10 monclient(hunting): authenticate will time out at 4542.110319s 2024-03-25T19:51:47.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.034+0000 7fc50ffff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc5101430c0 0x7fc51014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:47440/0 (socket says 172.21.15.81:47440) 2024-03-25T19:51:47.034 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.034+0000 7fc50ffff640 1 -- 172.21.15.81:0/3047003781 learned_addr learned my addr 172.21.15.81:0/3047003781 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:51:47.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.043+0000 7fc50e7fc640 1 -- 172.21.15.81:0/3047003781 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850113732 0 0) 0x7fc510142520 con 0x7fc5101430c0 2024-03-25T19:51:47.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.043+0000 7fc50e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:47.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.043+0000 7fc50e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:47.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.043+0000 7fc50e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:47.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.043+0000 7fc50e7fc640 10 monclient(hunting): my global_id is 4677 2024-03-25T19:51:47.042 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.043+0000 7fc50e7fc640 1 -- 172.21.15.81:0/3047003781 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4f8003520 con 0x7fc5101430c0 2024-03-25T19:51:47.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.051+0000 7fc50e7fc640 1 -- 172.21.15.81:0/3047003781 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1481122769 0 0) 0x7fc4f8003520 con 0x7fc5101430c0 2024-03-25T19:51:47.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.051+0000 7fc50e7fc640 1 -- 172.21.15.81:0/3047003781 >> v1:172.21.15.81:6790/0 conn(0x7fc51013f710 legacy=0x7fc5101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:51:47.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.051+0000 7fc50e7fc640 1 -- 172.21.15.81:0/3047003781 >> v1:172.21.15.98:6789/0 conn(0x7fc5100a98f0 legacy=0x7fc5100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:47.052 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.051+0000 7fc50e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:51:47.052 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.051+0000 7fc50e7fc640 1 monclient: found mon.noname-a 2024-03-25T19:51:47.052 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.051+0000 7fc50e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:47.052 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.051+0000 7fc50e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:51:47.052 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.051+0000 7fc50e7fc640 1 -- 172.21.15.81:0/3047003781 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5100b44c0 con 0x7fc5101430c0 2024-03-25T19:51:47.052 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.051+0000 7fc50e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:51:47.052 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.051+0000 7fc50e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:51:47.052 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.051+0000 7fc50e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:47.053 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.051+0000 7fc50e7fc640 1 -- 172.21.15.81:0/3047003781 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fc508001a10 con 0x7fc5101430c0 2024-03-25T19:51:47.053 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.051+0000 7fc50e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:47.053 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.051+0000 7fc50e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:47.053 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.051+0000 7fc50e7fc640 10 monclient: dump: 2024-03-25T19:51:47.053 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:47.053 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:47.053 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:47.053 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:47.053 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:47.053 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:47.053 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:47.053 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:47.053 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:47.053 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:47.053 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.051+0000 7fc515c86640 5 monclient: authenticate success, global_id 4677 2024-03-25T19:51:47.053 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.051+0000 7fc515c86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:47.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.081+0000 7fc50e7fc640 1 -- 172.21.15.81:0/3047003781 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc508001be0 con 0x7fc5101430c0 2024-03-25T19:51:47.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.081+0000 7fc50e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:47.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.082+0000 7fc50e7fc640 1 -- 172.21.15.81:0/3047003781 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fc508002030 con 0x7fc5101430c0 2024-03-25T19:51:47.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.082+0000 7fc515c86640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:47.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.082+0000 7fc515c86640 4 set_mon_vals no callback set 2024-03-25T19:51:47.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.082+0000 7fc50e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:47.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.082+0000 7fc50e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:47.082 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.082+0000 7fc50e7fc640 10 monclient: dump: 2024-03-25T19:51:47.082 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:47.082 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:47.082 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:47.082 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:47.082 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:47.082 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:47.082 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:47.082 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:47.082 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:47.082 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:47.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.082+0000 7fc515c86640 10 monclient: shutdown 2024-03-25T19:51:47.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.082+0000 7fc515c86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:47.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.082+0000 7fc515c86640 1 -- 172.21.15.81:0/3047003781 >> v1:172.21.15.81:6789/0 conn(0x7fc5101430c0 legacy=0x7fc51014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:47.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.082+0000 7fc515c86640 1 -- 172.21.15.81:0/3047003781 shutdown_connections 2024-03-25T19:51:47.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.082+0000 7fc515c86640 1 -- 172.21.15.81:0/3047003781 >> 172.21.15.81:0/3047003781 conn(0x7fc5100b7ea0 msgr2=0x7fc5100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:47.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.082+0000 7fc515c86640 1 -- 172.21.15.81:0/3047003781 shutdown_connections 2024-03-25T19:51:47.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.083+0000 7fc515c86640 1 -- 172.21.15.81:0/3047003781 wait complete. 2024-03-25T19:51:47.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.084+0000 7fc515c86640 10 monclient: build_initial_monmap 2024-03-25T19:51:47.084 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.084+0000 7fc515c86640 10 monclient: monmap: 2024-03-25T19:51:47.084 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:51:47.084 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:47.084 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:51:47.084 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:51:47.085 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:47.085 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:51:47.085 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:51:47.085 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:51:47.085 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:51:47.085 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:47.085 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.084+0000 7fc515c86640 1 librados: starting msgr at 2024-03-25T19:51:47.085 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.084+0000 7fc515c86640 1 librados: starting objecter 2024-03-25T19:51:47.085 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.084+0000 7fc515c86640 1 Processor -- start 2024-03-25T19:51:47.085 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.084+0000 7fc515c86640 1 -- start start 2024-03-25T19:51:47.085 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.084+0000 7fc515c86640 1 librados: setting wanted keys 2024-03-25T19:51:47.085 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.084+0000 7fc515c86640 1 librados: calling monclient init 2024-03-25T19:51:47.085 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.084+0000 7fc515c86640 10 monclient: init 2024-03-25T19:51:47.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.085+0000 7fc515c86640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:47.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.085+0000 7fc515c86640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:51:47.086 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.085+0000 7fc515c86640 10 monclient(hunting): picked mon.noname-c con 0x7fc5100a98f0 addr v1:172.21.15.81:6790/0 2024-03-25T19:51:47.087 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.085+0000 7fc515c86640 10 monclient(hunting): picked mon.noname-b con 0x7fc51013f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:51:47.087 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.085+0000 7fc515c86640 10 monclient(hunting): picked mon.noname-a con 0x7fc5101430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:51:47.087 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.085+0000 7fc515c86640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5101371e0 con 0x7fc5101430c0 2024-03-25T19:51:47.087 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.085+0000 7fc515c86640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5101374c0 con 0x7fc51013f710 2024-03-25T19:51:47.087 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.085+0000 7fc515c86640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc51013cb50 con 0x7fc5100a98f0 2024-03-25T19:51:47.087 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.085+0000 7fc515c86640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:47.087 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.085+0000 7fc515c86640 10 monclient(hunting): authenticate will time out at 4542.167125s 2024-03-25T19:51:47.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.089+0000 7fc50ffff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc5101430c0 0x7fc51013a970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:47466/0 (socket says 172.21.15.81:47466) 2024-03-25T19:51:47.089 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.089+0000 7fc50ffff640 1 -- 172.21.15.81:0/2969437767 learned_addr learned my addr 172.21.15.81:0/2969437767 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:51:47.090 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.091+0000 7fc4f7fff640 10 client.?.objecter ms_handle_connect 0x7fc5101430c0 2024-03-25T19:51:47.091 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.091+0000 7fc4f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:47.093 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.094+0000 7fc4f7fff640 10 client.?.objecter ms_handle_connect 0x7fc5100a98f0 2024-03-25T19:51:47.093 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.094+0000 7fc4f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:47.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.098+0000 7fc4f7fff640 10 client.?.objecter ms_handle_connect 0x7fc51013f710 2024-03-25T19:51:47.097 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.098+0000 7fc4f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:47.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.107+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093942853 0 0) 0x7fc5101371e0 con 0x7fc5101430c0 2024-03-25T19:51:47.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.107+0000 7fc4f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:47.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.107+0000 7fc4f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:47.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.107+0000 7fc4f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:47.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.107+0000 7fc4f7fff640 10 monclient(hunting): my global_id is 4680 2024-03-25T19:51:47.107 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.107+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4fc004eb0 con 0x7fc5101430c0 2024-03-25T19:51:47.109 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.109+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737191082 0 0) 0x7fc5101374c0 con 0x7fc51013f710 2024-03-25T19:51:47.109 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.109+0000 7fc4f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:47.109 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.109+0000 7fc4f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:47.109 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.109+0000 7fc4f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:47.109 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.109+0000 7fc4f7fff640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:51:47.109 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.109+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5101371e0 con 0x7fc51013f710 2024-03-25T19:51:47.111 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.112+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1141378548 0 0) 0x7fc51013cb50 con 0x7fc5100a98f0 2024-03-25T19:51:47.111 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.112+0000 7fc4f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:47.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.112+0000 7fc4f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:47.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.112+0000 7fc4f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:47.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.112+0000 7fc4f7fff640 10 monclient(hunting): my global_id is 4643 2024-03-25T19:51:47.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.112+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5101374c0 con 0x7fc5100a98f0 2024-03-25T19:51:47.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.121+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2753333382 0 0) 0x7fc5101371e0 con 0x7fc51013f710 2024-03-25T19:51:47.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.121+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc51013cb50 con 0x7fc51013f710 2024-03-25T19:51:47.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.121+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fc504001810 con 0x7fc51013f710 2024-03-25T19:51:47.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.121+0000 7fc4f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:47.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.121+0000 7fc4f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:47.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.121+0000 7fc4f7fff640 10 monclient(hunting): dump: 2024-03-25T19:51:47.121 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:47.121 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:47.121 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:47.122 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:47.122 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:47.122 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:47.122 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:47.122 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:47.122 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:47.122 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:47.122 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.121+0000 7fc4f7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:47.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 185557001 0 0) 0x7fc51013cb50 con 0x7fc51013f710 2024-03-25T19:51:47.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 >> v1:172.21.15.81:6790/0 conn(0x7fc5100a98f0 legacy=0x7fc5101378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:47.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 >> v1:172.21.15.81:6789/0 conn(0x7fc5101430c0 legacy=0x7fc51013a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:47.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc4f7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:51:47.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc4f7fff640 1 monclient: found mon.b 2024-03-25T19:51:47.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc4f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:47.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc4f7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:51:47.132 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51013b080 con 0x7fc51013f710 2024-03-25T19:51:47.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc4f7fff640 10 monclient: _finish_auth 0 2024-03-25T19:51:47.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc4f7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:51:47.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc4f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:47.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc515c86640 5 monclient: authenticate success, global_id 4660 2024-03-25T19:51:47.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc515c86640 10 monclient: _renew_subs 2024-03-25T19:51:47.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc515c86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:51:47.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc515c86640 1 -- 172.21.15.81:0/2969437767 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc510135d80 con 0x7fc51013f710 2024-03-25T19:51:47.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc515c86640 10 client.4660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:47.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc515c86640 10 monclient: _renew_subs 2024-03-25T19:51:47.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc515c86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:51:47.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc515c86640 1 -- 172.21.15.81:0/2969437767 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5101381d0 con 0x7fc51013f710 2024-03-25T19:51:47.133 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.132+0000 7fc515c86640 1 librados: init done 2024-03-25T19:51:47.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.133+0000 7fc515c86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:47.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.133+0000 7fc515c86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:47.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.133+0000 7fc515c86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:51:47.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.133+0000 7fc515c86640 1 -- 172.21.15.81:0/2969437767 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5100627b0 con 0x7fc51013f710 2024-03-25T19:51:47.150 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.150+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc504001d00 con 0x7fc51013f710 2024-03-25T19:51:47.150 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.150+0000 7fc4f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:47.150 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.150+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fc504003790 con 0x7fc51013f710 2024-03-25T19:51:47.150 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.150+0000 7fc4f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:47.150 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.150+0000 7fc4f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:47.150 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.150+0000 7fc4f7fff640 10 monclient: dump: 2024-03-25T19:51:47.151 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:47.151 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:47.151 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:47.151 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:47.151 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:47.151 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:47.151 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:47.151 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:47.151 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:47.151 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:47.151 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.150+0000 7fc50dffb640 4 set_mon_vals no callback set 2024-03-25T19:51:47.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.151+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fc504019760 con 0x7fc51013f710 2024-03-25T19:51:47.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.152+0000 7fc50f7fe640 10 monclient: get_auth_request con 0x7fc4fc06ddd0 auth_method 0 2024-03-25T19:51:47.152 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.152+0000 7fc4f7fff640 10 client.4660.objecter ms_handle_connect 0x7fc4fc06ddd0 2024-03-25T19:51:47.154 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.154+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fc504053510 con 0x7fc51013f710 2024-03-25T19:51:47.154 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.154+0000 7fc4f7fff640 10 client.4660.objecter ms_dispatch 0x7fc510000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:51:47.154 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.154+0000 7fc4f7fff640 3 client.4660.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:51:47.154 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.154+0000 7fc4f7fff640 3 client.4660.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:51:47.154 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.154+0000 7fc4f7fff640 20 client.4660.objecter dump_active .. 0 homeless 2024-03-25T19:51:47.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.408+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc50404ee50 con 0x7fc51013f710 2024-03-25T19:51:47.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.408+0000 7fc4f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:47.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.408+0000 7fc4f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:47.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.660+0000 7fc515c86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:47.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.660+0000 7fc515c86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:47.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.660+0000 7fc515c86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:51:47.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.660+0000 7fc515c86640 1 -- 172.21.15.81:0/2969437767 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5101362d0 con 0x7fc51013f710 2024-03-25T19:51:47.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.701+0000 7fc4f7fff640 1 -- 172.21.15.81:0/2969437767 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fc504050f50 con 0x7fc51013f710 2024-03-25T19:51:47.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.701+0000 7fc4f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:47.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.701+0000 7fc4f7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:47.701 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:51:47.701 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:47.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.703+0000 7fc515c86640 10 librados: watch_flush enter 2024-03-25T19:51:47.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.703+0000 7fc515c86640 10 librados: watch_flush exit 2024-03-25T19:51:47.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.703+0000 7fc515c86640 20 client.4660.objecter shutdown clearing up homeless session... 2024-03-25T19:51:47.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.703+0000 7fc515c86640 10 client.4660.objecter successfully canceled tick 2024-03-25T19:51:47.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.703+0000 7fc515c86640 1 -- 172.21.15.81:0/2969437767 >> v1:172.21.15.98:6800/3980829775 conn(0x7fc4fc06ddd0 legacy=0x7fc4fc070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:47.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.703+0000 7fc515c86640 10 monclient: shutdown 2024-03-25T19:51:47.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.703+0000 7fc515c86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:47.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.703+0000 7fc515c86640 1 -- 172.21.15.81:0/2969437767 >> v1:172.21.15.98:6789/0 conn(0x7fc51013f710 legacy=0x7fc51013a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:47.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.704+0000 7fc515c86640 1 -- 172.21.15.81:0/2969437767 shutdown_connections 2024-03-25T19:51:47.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.704+0000 7fc515c86640 1 -- 172.21.15.81:0/2969437767 >> 172.21.15.81:0/2969437767 conn(0x7fc5100b7ea0 msgr2=0x7fc510074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:47.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.704+0000 7fc515c86640 1 -- 172.21.15.81:0/2969437767 shutdown_connections 2024-03-25T19:51:47.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.704+0000 7fc515c86640 1 -- 172.21.15.81:0/2969437767 wait complete. 2024-03-25T19:51:47.704 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:47.704+0000 7fc515c86640 1 librados: shutdown 2024-03-25T19:51:47.715 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:54.521 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:54.623 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.623+0000 7fe3c88e4640 10 monclient: get_monmap_and_config 2024-03-25T19:51:54.623 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.623+0000 7fe3c88e4640 10 monclient: build_initial_monmap 2024-03-25T19:51:54.623 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.623+0000 7fe3c88e4640 10 monclient: monmap: 2024-03-25T19:51:54.623 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:51:54.623 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:54.623 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:51:54.624403+0000 2024-03-25T19:51:54.623 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:51:54.624403+0000 2024-03-25T19:51:54.623 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:54.623 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:54.623 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:51:54.623 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:51:54.624 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:51:54.624 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:54.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.624+0000 7fe3c88e4640 1 Processor -- start 2024-03-25T19:51:54.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.624+0000 7fe3c88e4640 1 -- start start 2024-03-25T19:51:54.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.624+0000 7fe3c88e4640 10 monclient: init 2024-03-25T19:51:54.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.624+0000 7fe3c88e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:54.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.624+0000 7fe3c88e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:54.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.624+0000 7fe3c88e4640 10 monclient(hunting): picked mon.noname-c con 0x7fe3c40b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:51:54.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.625+0000 7fe3c88e4640 10 monclient(hunting): picked mon.noname-a con 0x7fe3c40b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:51:54.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.625+0000 7fe3c88e4640 10 monclient(hunting): picked mon.noname-b con 0x7fe3c414be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:51:54.625 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.625+0000 7fe3c88e4640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c40af940 con 0x7fe3c40b1730 2024-03-25T19:51:54.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.625+0000 7fe3c88e4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c40b0a70 con 0x7fe3c40b4d60 2024-03-25T19:51:54.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.625+0000 7fe3c88e4640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c40b0d50 con 0x7fe3c414be90 2024-03-25T19:51:54.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.625+0000 7fe3c88e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:54.626 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.625+0000 7fe3c88e4640 10 monclient(hunting): authenticate will time out at 4549.706897s 2024-03-25T19:51:54.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.629+0000 7fe3c1d9b640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe3c40b1730 0x7fe3c40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:40280/0 (socket says 172.21.15.81:40280) 2024-03-25T19:51:54.628 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.629+0000 7fe3c1d9b640 1 -- 172.21.15.81:0/210566001 learned_addr learned my addr 172.21.15.81:0/210566001 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:51:54.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.636+0000 7fe3c159a640 1 -- 172.21.15.81:0/210566001 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077882559 0 0) 0x7fe3c40af940 con 0x7fe3c40b1730 2024-03-25T19:51:54.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.636+0000 7fe3c159a640 10 monclient(hunting): _negotiate 2024-03-25T19:51:54.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.636+0000 7fe3c159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:54.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.636+0000 7fe3c159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:54.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.636+0000 7fe3c159a640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:51:54.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.636+0000 7fe3c159a640 1 -- 172.21.15.81:0/210566001 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3ac003520 con 0x7fe3c40b1730 2024-03-25T19:51:54.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.646+0000 7fe3c159a640 1 -- 172.21.15.81:0/210566001 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1835421254 0 0) 0x7fe3ac003520 con 0x7fe3c40b1730 2024-03-25T19:51:54.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.646+0000 7fe3c159a640 1 -- 172.21.15.81:0/210566001 >> v1:172.21.15.81:6790/0 conn(0x7fe3c414be90 legacy=0x7fe3c414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:54.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.646+0000 7fe3c159a640 1 -- 172.21.15.81:0/210566001 >> v1:172.21.15.98:6789/0 conn(0x7fe3c40b4d60 legacy=0x7fe3c413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:51:54.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.646+0000 7fe3c159a640 10 monclient: _finish_hunting 0 2024-03-25T19:51:54.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.646+0000 7fe3c159a640 1 monclient: found mon.noname-a 2024-03-25T19:51:54.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.646+0000 7fe3c159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:54.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.646+0000 7fe3c159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:51:54.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.646+0000 7fe3c159a640 1 -- 172.21.15.81:0/210566001 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c40b1030 con 0x7fe3c40b1730 2024-03-25T19:51:54.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.646+0000 7fe3c159a640 10 monclient: _finish_auth 0 2024-03-25T19:51:54.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.646+0000 7fe3c159a640 10 monclient: _check_auth_tickets 2024-03-25T19:51:54.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.646+0000 7fe3c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:54.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.646+0000 7fe3c159a640 1 -- 172.21.15.81:0/210566001 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe3b8001690 con 0x7fe3c40b1730 2024-03-25T19:51:54.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.646+0000 7fe3c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:54.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.646+0000 7fe3c159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:54.647 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.646+0000 7fe3c159a640 10 monclient: dump: 2024-03-25T19:51:54.647 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:54.647 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:54.647 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:54.647 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:54.647 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:54.647 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:54.648 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:54.648 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:54.648 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:54.648 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:54.648 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.646+0000 7fe3c88e4640 5 monclient: authenticate success, global_id 4683 2024-03-25T19:51:54.648 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.646+0000 7fe3c88e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:54.655 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.656+0000 7fe3c159a640 1 -- 172.21.15.81:0/210566001 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3b8001860 con 0x7fe3c40b1730 2024-03-25T19:51:54.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.656+0000 7fe3c159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:54.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.656+0000 7fe3c159a640 1 -- 172.21.15.81:0/210566001 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe3b8001c70 con 0x7fe3c40b1730 2024-03-25T19:51:54.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.656+0000 7fe3c88e4640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:54.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.656+0000 7fe3c88e4640 4 set_mon_vals no callback set 2024-03-25T19:51:54.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.656+0000 7fe3c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:54.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.656+0000 7fe3c159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:54.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.656+0000 7fe3c159a640 10 monclient: dump: 2024-03-25T19:51:54.656 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:54.656 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:54.656 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:54.656 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:54.656 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:54.656 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:54.657 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:54.657 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:54.657 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:54.657 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:54.658 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.657+0000 7fe3c88e4640 10 monclient: shutdown 2024-03-25T19:51:54.658 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.657+0000 7fe3c88e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:54.658 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.657+0000 7fe3c88e4640 1 -- 172.21.15.81:0/210566001 >> v1:172.21.15.81:6789/0 conn(0x7fe3c40b1730 legacy=0x7fe3c40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:54.658 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.657+0000 7fe3c88e4640 1 -- 172.21.15.81:0/210566001 shutdown_connections 2024-03-25T19:51:54.658 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.657+0000 7fe3c88e4640 1 -- 172.21.15.81:0/210566001 >> 172.21.15.81:0/210566001 conn(0x7fe3c40617a0 msgr2=0x7fe3c4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:54.658 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.657+0000 7fe3c88e4640 1 -- 172.21.15.81:0/210566001 shutdown_connections 2024-03-25T19:51:54.658 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.657+0000 7fe3c88e4640 1 -- 172.21.15.81:0/210566001 wait complete. 2024-03-25T19:51:54.658 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.658+0000 7fe3c88e4640 10 monclient: build_initial_monmap 2024-03-25T19:51:54.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.658+0000 7fe3c88e4640 10 monclient: monmap: 2024-03-25T19:51:54.659 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:51:54.659 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:54.659 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:51:54.659 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:51:54.659 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:54.659 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:51:54.659 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:51:54.659 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:51:54.659 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:51:54.659 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:54.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.658+0000 7fe3c88e4640 1 librados: starting msgr at 2024-03-25T19:51:54.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.658+0000 7fe3c88e4640 1 librados: starting objecter 2024-03-25T19:51:54.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.658+0000 7fe3c88e4640 1 Processor -- start 2024-03-25T19:51:54.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.658+0000 7fe3c88e4640 1 -- start start 2024-03-25T19:51:54.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.658+0000 7fe3c88e4640 1 librados: setting wanted keys 2024-03-25T19:51:54.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.658+0000 7fe3c88e4640 1 librados: calling monclient init 2024-03-25T19:51:54.660 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.658+0000 7fe3c88e4640 10 monclient: init 2024-03-25T19:51:54.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.659+0000 7fe3c88e4640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:54.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.659+0000 7fe3c88e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:54.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.659+0000 7fe3c88e4640 10 monclient(hunting): picked mon.noname-c con 0x7fe3c40b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:51:54.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.659+0000 7fe3c88e4640 10 monclient(hunting): picked mon.noname-a con 0x7fe3c40b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:51:54.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.659+0000 7fe3c88e4640 10 monclient(hunting): picked mon.noname-b con 0x7fe3c414be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:51:54.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.659+0000 7fe3c88e4640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c4144160 con 0x7fe3c40b4d60 2024-03-25T19:51:54.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.659+0000 7fe3c88e4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c413e880 con 0x7fe3c414be90 2024-03-25T19:51:54.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.659+0000 7fe3c88e4640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c413ea10 con 0x7fe3c40b1730 2024-03-25T19:51:54.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.659+0000 7fe3c88e4640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:54.661 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.659+0000 7fe3c88e4640 10 monclient(hunting): authenticate will time out at 4549.741245s 2024-03-25T19:51:54.671 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.671+0000 7fe3c1d9b640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fe3c40b4d60 0x7fe3c412fcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:40320/0 (socket says 172.21.15.81:40320) 2024-03-25T19:51:54.671 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.671+0000 7fe3c1d9b640 1 -- 172.21.15.81:0/538799533 learned_addr learned my addr 172.21.15.81:0/538799533 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:51:54.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.672+0000 7fe3b2ffd640 10 client.?.objecter ms_handle_connect 0x7fe3c40b4d60 2024-03-25T19:51:54.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.672+0000 7fe3b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:54.681 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.682+0000 7fe3b2ffd640 10 client.?.objecter ms_handle_connect 0x7fe3c414be90 2024-03-25T19:51:54.681 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.682+0000 7fe3b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:54.685 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.685+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335676976 0 0) 0x7fe3c4144160 con 0x7fe3c40b4d60 2024-03-25T19:51:54.685 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.686+0000 7fe3b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:54.685 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.686+0000 7fe3b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:54.686 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.686+0000 7fe3b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:54.686 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.686+0000 7fe3b2ffd640 10 monclient(hunting): my global_id is 4686 2024-03-25T19:51:54.686 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.686+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a4004eb0 con 0x7fe3c40b4d60 2024-03-25T19:51:54.686 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.686+0000 7fe3b2ffd640 10 client.?.objecter ms_handle_connect 0x7fe3c40b1730 2024-03-25T19:51:54.686 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.686+0000 7fe3b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:54.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.696+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393798434 0 0) 0x7fe3c413e880 con 0x7fe3c414be90 2024-03-25T19:51:54.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.696+0000 7fe3b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:54.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.696+0000 7fe3b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:54.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.696+0000 7fe3b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:54.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.696+0000 7fe3b2ffd640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:51:54.696 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.696+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3c4144160 con 0x7fe3c414be90 2024-03-25T19:51:54.698 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.699+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 741684745 0 0) 0x7fe3a4004eb0 con 0x7fe3c40b4d60 2024-03-25T19:51:54.698 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.699+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3c413e880 con 0x7fe3c40b4d60 2024-03-25T19:51:54.699 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.699+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe3b8001b70 con 0x7fe3c40b4d60 2024-03-25T19:51:54.699 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.699+0000 7fe3b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:54.699 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.699+0000 7fe3b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:54.699 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.699+0000 7fe3b2ffd640 10 monclient(hunting): dump: 2024-03-25T19:51:54.699 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:54.699 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:54.699 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:54.699 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:54.699 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:54.699 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:54.699 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:54.699 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:54.699 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:54.699 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:54.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.699+0000 7fe3b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:54.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.701+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589564139 0 0) 0x7fe3c413ea10 con 0x7fe3c40b1730 2024-03-25T19:51:54.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.701+0000 7fe3b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:54.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.701+0000 7fe3b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:54.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.701+0000 7fe3b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:54.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.701+0000 7fe3b2ffd640 10 monclient(hunting): my global_id is 4649 2024-03-25T19:51:54.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.701+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a4004eb0 con 0x7fe3c40b1730 2024-03-25T19:51:54.706 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.706+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1865759601 0 0) 0x7fe3c413e880 con 0x7fe3c40b4d60 2024-03-25T19:51:54.707 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.706+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 >> v1:172.21.15.81:6790/0 conn(0x7fe3c40b1730 legacy=0x7fe3c4143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:54.707 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.706+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 >> v1:172.21.15.98:6789/0 conn(0x7fe3c414be90 legacy=0x7fe3c413d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:54.707 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.706+0000 7fe3b2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:54.707 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.706+0000 7fe3b2ffd640 1 monclient: found mon.a 2024-03-25T19:51:54.707 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.706+0000 7fe3b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:54.707 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.706+0000 7fe3b2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:51:54.707 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.706+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c4136a80 con 0x7fe3c40b4d60 2024-03-25T19:51:54.707 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.706+0000 7fe3b2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:51:54.707 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.706+0000 7fe3b2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:54.707 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.706+0000 7fe3b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:54.707 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.706+0000 7fe3c88e4640 5 monclient: authenticate success, global_id 4686 2024-03-25T19:51:54.707 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.706+0000 7fe3c88e4640 10 monclient: _renew_subs 2024-03-25T19:51:54.708 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.706+0000 7fe3c88e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:51:54.708 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.706+0000 7fe3c88e4640 1 -- 172.21.15.81:0/538799533 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c4136cd0 con 0x7fe3c40b4d60 2024-03-25T19:51:54.708 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.707+0000 7fe3c88e4640 10 client.4686.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:54.708 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.707+0000 7fe3c88e4640 10 monclient: _renew_subs 2024-03-25T19:51:54.708 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.707+0000 7fe3c88e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:51:54.708 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.707+0000 7fe3c88e4640 1 -- 172.21.15.81:0/538799533 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c41305c0 con 0x7fe3c40b4d60 2024-03-25T19:51:54.708 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.707+0000 7fe3c88e4640 1 librados: init done 2024-03-25T19:51:54.709 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.708+0000 7fe3c88e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:54.709 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.708+0000 7fe3c88e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:54.709 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.708+0000 7fe3c88e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:51:54.709 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:54.708+0000 7fe3c88e4640 1 -- 172.21.15.81:0/538799533 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c41442f0 con 0x7fe3c40b4d60 2024-03-25T19:51:55.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.013+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3b8004bf0 con 0x7fe3c40b4d60 2024-03-25T19:51:55.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.013+0000 7fe3b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:55.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.013+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fe3b8004f00 con 0x7fe3c40b4d60 2024-03-25T19:51:55.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.013+0000 7fe3b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:55.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.013+0000 7fe3b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:55.013 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.013+0000 7fe3b2ffd640 10 monclient: dump: 2024-03-25T19:51:55.013 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:51:55.013 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:51:55.013 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:55.013 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:51:55.013 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:55.014 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:51:55.014 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:51:55.014 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:51:55.014 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:51:55.014 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:51:55.014 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.013+0000 7fe3c0d99640 4 set_mon_vals no callback set 2024-03-25T19:51:55.015 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.014+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fe3b801aed0 con 0x7fe3c40b4d60 2024-03-25T19:51:55.015 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.014+0000 7fe3c259c640 10 monclient: get_auth_request con 0x7fe3a406ddb0 auth_method 0 2024-03-25T19:51:55.015 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.015+0000 7fe3b2ffd640 10 client.4686.objecter ms_handle_connect 0x7fe3a406ddb0 2024-03-25T19:51:55.020 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.020+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7fe3b8054970 con 0x7fe3c40b4d60 2024-03-25T19:51:55.020 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.020+0000 7fe3b2ffd640 10 client.4686.objecter ms_dispatch 0x7fe3c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:51:55.020 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.020+0000 7fe3b2ffd640 3 client.4686.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:51:55.020 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.020+0000 7fe3b2ffd640 3 client.4686.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:51:55.020 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.020+0000 7fe3b2ffd640 20 client.4686.objecter dump_active .. 0 homeless 2024-03-25T19:51:55.020 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.020+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe3b8050480 con 0x7fe3c40b4d60 2024-03-25T19:51:55.021 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.020+0000 7fe3b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:55.021 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.020+0000 7fe3b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:55.278 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.279+0000 7fe3c88e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:55.279 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.279+0000 7fe3c88e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:55.279 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.279+0000 7fe3c88e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:51:55.279 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.279+0000 7fe3c88e4640 1 -- 172.21.15.81:0/538799533 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3c40b2270 con 0x7fe3c40b4d60 2024-03-25T19:51:55.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.314+0000 7fe3b2ffd640 1 -- 172.21.15.81:0/538799533 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7fe3b80523b0 con 0x7fe3c40b4d60 2024-03-25T19:51:55.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.314+0000 7fe3b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:55.314 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.314+0000 7fe3b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:55.314 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:51:55.315 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:55.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.316+0000 7fe3c88e4640 10 librados: watch_flush enter 2024-03-25T19:51:55.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.317+0000 7fe3c88e4640 10 librados: watch_flush exit 2024-03-25T19:51:55.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.317+0000 7fe3c88e4640 20 client.4686.objecter shutdown clearing up homeless session... 2024-03-25T19:51:55.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.317+0000 7fe3c88e4640 10 client.4686.objecter successfully canceled tick 2024-03-25T19:51:55.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.317+0000 7fe3c88e4640 1 -- 172.21.15.81:0/538799533 >> v1:172.21.15.98:6800/3980829775 conn(0x7fe3a406ddb0 legacy=0x7fe3a4070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:55.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.317+0000 7fe3c88e4640 10 monclient: shutdown 2024-03-25T19:51:55.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.317+0000 7fe3c88e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:55.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.317+0000 7fe3c88e4640 1 -- 172.21.15.81:0/538799533 >> v1:172.21.15.81:6789/0 conn(0x7fe3c40b4d60 legacy=0x7fe3c412fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:55.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.317+0000 7fe3c88e4640 1 -- 172.21.15.81:0/538799533 shutdown_connections 2024-03-25T19:51:55.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.317+0000 7fe3c88e4640 1 -- 172.21.15.81:0/538799533 >> 172.21.15.81:0/538799533 conn(0x7fe3c40617a0 msgr2=0x7fe3c40b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:55.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.317+0000 7fe3c88e4640 1 -- 172.21.15.81:0/538799533 shutdown_connections 2024-03-25T19:51:55.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.317+0000 7fe3c88e4640 1 -- 172.21.15.81:0/538799533 wait complete. 2024-03-25T19:51:55.318 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:51:55.317+0000 7fe3c88e4640 1 librados: shutdown 2024-03-25T19:51:55.328 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:02.135 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:02.250 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.250+0000 7f0515cc1640 10 monclient: get_monmap_and_config 2024-03-25T19:52:02.250 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.250+0000 7f0515cc1640 10 monclient: build_initial_monmap 2024-03-25T19:52:02.250 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.250+0000 7f0515cc1640 10 monclient: monmap: 2024-03-25T19:52:02.250 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:52:02.250 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:02.251 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:52:02.251937+0000 2024-03-25T19:52:02.251 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:52:02.251937+0000 2024-03-25T19:52:02.251 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:02.251 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:02.251 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:52:02.251 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:52:02.251 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:52:02.251 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:02.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.250+0000 7f0515cc1640 1 Processor -- start 2024-03-25T19:52:02.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.250+0000 7f0515cc1640 1 -- start start 2024-03-25T19:52:02.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.250+0000 7f0515cc1640 10 monclient: init 2024-03-25T19:52:02.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.251+0000 7f0515cc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:02.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.251+0000 7f0515cc1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:52:02.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.251+0000 7f0515cc1640 10 monclient(hunting): picked mon.noname-c con 0x7f05100b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:52:02.252 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.251+0000 7f0515cc1640 10 monclient(hunting): picked mon.noname-a con 0x7f05100b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:52:02.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.251+0000 7f0515cc1640 10 monclient(hunting): picked mon.noname-b con 0x7f051014be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:52:02.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.251+0000 7f0515cc1640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05100af940 con 0x7f05100b1730 2024-03-25T19:52:02.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.251+0000 7f0515cc1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05100b0a70 con 0x7f05100b4d60 2024-03-25T19:52:02.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.251+0000 7f0515cc1640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05100b0d50 con 0x7f051014be90 2024-03-25T19:52:02.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.251+0000 7f0515cc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:02.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.251+0000 7f0515cc1640 10 monclient(hunting): authenticate will time out at 4557.334384s 2024-03-25T19:52:02.264 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.264+0000 7f050ffff640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f051014be90 0x7f051014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:55256/0 (socket says 172.21.15.81:55256) 2024-03-25T19:52:02.264 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.264+0000 7f050ffff640 1 -- 172.21.15.81:0/1436226510 learned_addr learned my addr 172.21.15.81:0/1436226510 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:52:02.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.280+0000 7f050e7fc640 1 -- 172.21.15.81:0/1436226510 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598794045 0 0) 0x7f05100b0d50 con 0x7f051014be90 2024-03-25T19:52:02.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.280+0000 7f050e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:02.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.280+0000 7f050e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:02.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.280+0000 7f050e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:02.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.280+0000 7f050e7fc640 10 monclient(hunting): my global_id is 4652 2024-03-25T19:52:02.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.280+0000 7f050e7fc640 1 -- 172.21.15.81:0/1436226510 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f8003520 con 0x7f051014be90 2024-03-25T19:52:02.285 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.284+0000 7f050e7fc640 1 -- 172.21.15.81:0/1436226510 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880417180 0 0) 0x7f05100af940 con 0x7f05100b1730 2024-03-25T19:52:02.285 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.284+0000 7f050e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:02.285 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.284+0000 7f050e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:02.285 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.284+0000 7f050e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:02.285 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.284+0000 7f050e7fc640 10 monclient(hunting): my global_id is 4689 2024-03-25T19:52:02.285 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.284+0000 7f050e7fc640 1 -- 172.21.15.81:0/1436226510 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05100b0d50 con 0x7f05100b1730 2024-03-25T19:52:02.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.286+0000 7f050e7fc640 1 -- 172.21.15.81:0/1436226510 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161486502 0 0) 0x7f05100b0a70 con 0x7f05100b4d60 2024-03-25T19:52:02.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.286+0000 7f050e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:02.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.286+0000 7f050e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:02.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.286+0000 7f050e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:02.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.286+0000 7f050e7fc640 10 monclient(hunting): my global_id is 4666 2024-03-25T19:52:02.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.286+0000 7f050e7fc640 1 -- 172.21.15.81:0/1436226510 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05100af940 con 0x7f05100b4d60 2024-03-25T19:52:02.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.304+0000 7f050e7fc640 1 -- 172.21.15.81:0/1436226510 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3995702109 0 0) 0x7f04f8003520 con 0x7f051014be90 2024-03-25T19:52:02.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.304+0000 7f050e7fc640 1 -- 172.21.15.81:0/1436226510 >> v1:172.21.15.98:6789/0 conn(0x7f05100b4d60 legacy=0x7f051013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:02.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.304+0000 7f050e7fc640 1 -- 172.21.15.81:0/1436226510 >> v1:172.21.15.81:6789/0 conn(0x7f05100b1730 legacy=0x7f05100b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:02.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.304+0000 7f050e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:52:02.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.304+0000 7f050e7fc640 1 monclient: found mon.noname-b 2024-03-25T19:52:02.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.304+0000 7f050e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:02.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.304+0000 7f050e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:52:02.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.304+0000 7f050e7fc640 1 -- 172.21.15.81:0/1436226510 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05100b1030 con 0x7f051014be90 2024-03-25T19:52:02.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.304+0000 7f050e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:52:02.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.304+0000 7f050e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:52:02.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.304+0000 7f050e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:02.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.304+0000 7f050e7fc640 1 -- 172.21.15.81:0/1436226510 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f0508001a10 con 0x7f051014be90 2024-03-25T19:52:02.305 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.304+0000 7f0515cc1640 5 monclient: authenticate success, global_id 4652 2024-03-25T19:52:02.306 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.304+0000 7f050e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:02.306 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.304+0000 7f050e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:02.306 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.304+0000 7f050e7fc640 10 monclient: dump: 2024-03-25T19:52:02.306 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:02.306 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:02.306 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:02.306 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:02.306 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:02.306 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:02.306 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:02.306 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:02.306 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:02.306 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:02.306 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.304+0000 7f0515cc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:02.337 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.336+0000 7f050e7fc640 1 -- 172.21.15.81:0/1436226510 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0508001b70 con 0x7f051014be90 2024-03-25T19:52:02.337 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.336+0000 7f050e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:02.337 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.336+0000 7f050e7fc640 1 -- 172.21.15.81:0/1436226510 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f0508001f80 con 0x7f051014be90 2024-03-25T19:52:02.337 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.336+0000 7f0515cc1640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:02.337 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.336+0000 7f0515cc1640 4 set_mon_vals no callback set 2024-03-25T19:52:02.337 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.336+0000 7f050e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:02.337 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.336+0000 7f050e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:52:02.338 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.336+0000 7f050e7fc640 10 monclient: dump: 2024-03-25T19:52:02.338 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:02.338 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:02.338 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:02.338 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:02.338 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:02.338 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:02.338 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:02.338 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:02.338 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:02.338 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:02.339 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.337+0000 7f0515cc1640 10 monclient: shutdown 2024-03-25T19:52:02.339 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.337+0000 7f0515cc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:02.339 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.337+0000 7f0515cc1640 1 -- 172.21.15.81:0/1436226510 >> v1:172.21.15.81:6790/0 conn(0x7f051014be90 legacy=0x7f051014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:02.340 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.337+0000 7f0515cc1640 1 -- 172.21.15.81:0/1436226510 shutdown_connections 2024-03-25T19:52:02.340 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.337+0000 7f0515cc1640 1 -- 172.21.15.81:0/1436226510 >> 172.21.15.81:0/1436226510 conn(0x7f05100617a0 msgr2=0x7f0510061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:02.340 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.337+0000 7f0515cc1640 1 -- 172.21.15.81:0/1436226510 shutdown_connections 2024-03-25T19:52:02.340 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.337+0000 7f0515cc1640 1 -- 172.21.15.81:0/1436226510 wait complete. 2024-03-25T19:52:02.340 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.338+0000 7f0515cc1640 10 monclient: build_initial_monmap 2024-03-25T19:52:02.340 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.338+0000 7f0515cc1640 10 monclient: monmap: 2024-03-25T19:52:02.340 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:52:02.340 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:02.340 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:52:02.340 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:52:02.340 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:02.340 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:52:02.340 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:52:02.340 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:52:02.340 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:52:02.341 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:02.341 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.338+0000 7f0515cc1640 1 librados: starting msgr at 2024-03-25T19:52:02.341 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.338+0000 7f0515cc1640 1 librados: starting objecter 2024-03-25T19:52:02.341 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.338+0000 7f0515cc1640 1 Processor -- start 2024-03-25T19:52:02.341 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.339+0000 7f0515cc1640 1 -- start start 2024-03-25T19:52:02.341 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.339+0000 7f0515cc1640 1 librados: setting wanted keys 2024-03-25T19:52:02.341 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.339+0000 7f0515cc1640 1 librados: calling monclient init 2024-03-25T19:52:02.341 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.339+0000 7f0515cc1640 10 monclient: init 2024-03-25T19:52:02.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.339+0000 7f0515cc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:02.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.339+0000 7f0515cc1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:52:02.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.339+0000 7f0515cc1640 10 monclient(hunting): picked mon.noname-c con 0x7f05100b1730 addr v1:172.21.15.81:6790/0 2024-03-25T19:52:02.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.339+0000 7f0515cc1640 10 monclient(hunting): picked mon.noname-a con 0x7f05100b4d60 addr v1:172.21.15.81:6789/0 2024-03-25T19:52:02.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.339+0000 7f0515cc1640 10 monclient(hunting): picked mon.noname-b con 0x7f051014be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:52:02.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.339+0000 7f0515cc1640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0510137d30 con 0x7f05100b4d60 2024-03-25T19:52:02.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.339+0000 7f0515cc1640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0510137f80 con 0x7f051014be90 2024-03-25T19:52:02.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.339+0000 7f0515cc1640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05101374c0 con 0x7f05100b1730 2024-03-25T19:52:02.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.339+0000 7f0515cc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:02.342 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.339+0000 7f0515cc1640 10 monclient(hunting): authenticate will time out at 4557.422591s 2024-03-25T19:52:02.352 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.351+0000 7f050effd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f05100b4d60 0x7f05101dab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:51974/0 (socket says 172.21.15.81:51974) 2024-03-25T19:52:02.352 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.351+0000 7f050f7fe640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f05100b1730 0x7f0510134f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:55336/0 (socket says 172.21.15.81:55336) 2024-03-25T19:52:02.352 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.351+0000 7f050effd640 1 -- 172.21.15.81:0/2743506765 learned_addr learned my addr 172.21.15.81:0/2743506765 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:52:02.354 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.353+0000 7f04f7fff640 10 client.?.objecter ms_handle_connect 0x7f051014be90 2024-03-25T19:52:02.354 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.354+0000 7f04f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:02.355 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.354+0000 7f04f7fff640 10 client.?.objecter ms_handle_connect 0x7f05100b4d60 2024-03-25T19:52:02.355 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.354+0000 7f04f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:02.355 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.355+0000 7f04f7fff640 10 client.?.objecter ms_handle_connect 0x7f05100b1730 2024-03-25T19:52:02.355 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.355+0000 7f04f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:02.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.362+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726592486 0 0) 0x7f0510137f80 con 0x7f051014be90 2024-03-25T19:52:02.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.362+0000 7f04f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:02.362 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.362+0000 7f04f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:02.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.362+0000 7f04f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:02.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.362+0000 7f04f7fff640 10 monclient(hunting): my global_id is 4669 2024-03-25T19:52:02.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.362+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04fc004eb0 con 0x7f051014be90 2024-03-25T19:52:02.370 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.369+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318790800 0 0) 0x7f0510137d30 con 0x7f05100b4d60 2024-03-25T19:52:02.370 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.369+0000 7f04f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:02.370 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.369+0000 7f04f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:02.370 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.370+0000 7f04f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:02.370 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.370+0000 7f04f7fff640 10 monclient(hunting): my global_id is 4692 2024-03-25T19:52:02.371 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.370+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0510137f80 con 0x7f05100b4d60 2024-03-25T19:52:02.374 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.374+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360376304 0 0) 0x7f05101374c0 con 0x7f05100b1730 2024-03-25T19:52:02.374 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.374+0000 7f04f7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:02.374 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.374+0000 7f04f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:02.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.374+0000 7f04f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:02.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.374+0000 7f04f7fff640 10 monclient(hunting): my global_id is 4655 2024-03-25T19:52:02.375 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.374+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0510137d30 con 0x7f05100b1730 2024-03-25T19:52:02.391 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.390+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2057399522 0 0) 0x7f04fc004eb0 con 0x7f051014be90 2024-03-25T19:52:02.391 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.390+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05101374c0 con 0x7f051014be90 2024-03-25T19:52:02.391 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.390+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f0508001e80 con 0x7f051014be90 2024-03-25T19:52:02.391 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.390+0000 7f04f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:02.391 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.390+0000 7f04f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:02.391 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.390+0000 7f04f7fff640 10 monclient(hunting): dump: 2024-03-25T19:52:02.392 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:02.392 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:02.392 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:02.392 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:02.392 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:02.392 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:02.392 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:02.392 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:02.392 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:02.392 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:02.392 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.390+0000 7f04f7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:02.396 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.396+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 314938593 0 0) 0x7f0510137f80 con 0x7f05100b4d60 2024-03-25T19:52:02.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.396+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04fc004eb0 con 0x7f05100b4d60 2024-03-25T19:52:02.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.396+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f05040019b0 con 0x7f05100b4d60 2024-03-25T19:52:02.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.396+0000 7f04f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:02.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.396+0000 7f04f7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:02.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.396+0000 7f04f7fff640 10 monclient(hunting): dump: 2024-03-25T19:52:02.397 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:02.397 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:02.397 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:02.397 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:02.397 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:02.397 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:02.397 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:02.397 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:02.397 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:02.398 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:02.398 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.396+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4154857890 0 0) 0x7f0510137d30 con 0x7f05100b1730 2024-03-25T19:52:02.398 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.396+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0510137f80 con 0x7f05100b1730 2024-03-25T19:52:02.398 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.396+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f05000018f0 con 0x7f05100b1730 2024-03-25T19:52:02.398 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.396+0000 7f04f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:02.398 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.396+0000 7f04f7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:52:02.398 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.396+0000 7f04f7fff640 10 monclient(hunting): dump: 2024-03-25T19:52:02.398 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:02.398 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:02.398 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:02.398 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:02.398 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:02.398 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:02.398 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:02.399 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:02.399 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:02.399 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:02.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.415+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4293614353 0 0) 0x7f05101374c0 con 0x7f051014be90 2024-03-25T19:52:02.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.415+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 >> v1:172.21.15.81:6790/0 conn(0x7f05100b1730 legacy=0x7f0510134f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:02.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.415+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 >> v1:172.21.15.81:6789/0 conn(0x7f05100b4d60 legacy=0x7f05101dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:02.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.415+0000 7f04f7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:52:02.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.415+0000 7f04f7fff640 1 monclient: found mon.b 2024-03-25T19:52:02.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.415+0000 7f04f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:02.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.415+0000 7f04f7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:02.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.415+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0510135650 con 0x7f051014be90 2024-03-25T19:52:02.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.415+0000 7f04f7fff640 10 monclient: _finish_auth 0 2024-03-25T19:52:02.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.415+0000 7f04f7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:52:02.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.416+0000 7f04f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:02.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.416+0000 7f0515cc1640 5 monclient: authenticate success, global_id 4669 2024-03-25T19:52:02.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.416+0000 7f0515cc1640 10 monclient: _renew_subs 2024-03-25T19:52:02.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.416+0000 7f0515cc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:02.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.416+0000 7f0515cc1640 1 -- 172.21.15.81:0/2743506765 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0510144e60 con 0x7f051014be90 2024-03-25T19:52:02.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.416+0000 7f0515cc1640 10 client.4669.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:02.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.416+0000 7f0515cc1640 10 monclient: _renew_subs 2024-03-25T19:52:02.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.416+0000 7f0515cc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:02.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.416+0000 7f0515cc1640 1 -- 172.21.15.81:0/2743506765 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0510140380 con 0x7f051014be90 2024-03-25T19:52:02.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.416+0000 7f0515cc1640 1 librados: init done 2024-03-25T19:52:02.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.417+0000 7f0515cc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:02.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.417+0000 7f0515cc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:02.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.417+0000 7f0515cc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:02.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.417+0000 7f0515cc1640 1 -- 172.21.15.81:0/2743506765 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0510141a10 con 0x7f051014be90 2024-03-25T19:52:02.714 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.713+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0508002020 con 0x7f051014be90 2024-03-25T19:52:02.714 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.713+0000 7f04f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:02.714 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.713+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f0508004fb0 con 0x7f051014be90 2024-03-25T19:52:02.714 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.713+0000 7f04f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:02.714 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.713+0000 7f04f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:02.714 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.713+0000 7f04f7fff640 10 monclient: dump: 2024-03-25T19:52:02.714 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:02.714 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:02.714 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:02.714 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:02.715 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:02.715 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:02.715 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:02.715 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:02.715 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:02.715 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:02.715 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.713+0000 7f050dffb640 4 set_mon_vals no callback set 2024-03-25T19:52:02.715 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.714+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f050801af80 con 0x7f051014be90 2024-03-25T19:52:02.716 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.714+0000 7f050f7fe640 10 monclient: get_auth_request con 0x7f04fc06e0f0 auth_method 0 2024-03-25T19:52:02.716 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.715+0000 7f04f7fff640 10 client.4669.objecter ms_handle_connect 0x7f04fc06e0f0 2024-03-25T19:52:02.719 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.718+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1672310569 0 0) 0x7f0508054a30 con 0x7f051014be90 2024-03-25T19:52:02.719 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.718+0000 7f04f7fff640 10 client.4669.objecter ms_dispatch 0x7f0510000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:52:02.719 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.718+0000 7f04f7fff640 3 client.4669.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:52:02.719 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.718+0000 7f04f7fff640 3 client.4669.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:52:02.719 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.718+0000 7f04f7fff640 20 client.4669.objecter dump_active .. 0 homeless 2024-03-25T19:52:02.719 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.719+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0508050540 con 0x7f051014be90 2024-03-25T19:52:02.720 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.719+0000 7f04f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:02.720 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.719+0000 7f04f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:02.951 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.950+0000 7f0515cc1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:02.951 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.950+0000 7f0515cc1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:02.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.950+0000 7f0515cc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:02.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.950+0000 7f0515cc1640 1 -- 172.21.15.81:0/2743506765 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0510138110 con 0x7f051014be90 2024-03-25T19:52:02.987 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.986+0000 7f04f7fff640 1 -- 172.21.15.81:0/2743506765 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 885252940) 0x7f0508052470 con 0x7f051014be90 2024-03-25T19:52:02.987 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.986+0000 7f04f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:02.987 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.986+0000 7f04f7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:02.987 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:52:02.987 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":10,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:42:05.325115+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:42:05.325115+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:02.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.988+0000 7f0515cc1640 10 librados: watch_flush enter 2024-03-25T19:52:02.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.988+0000 7f0515cc1640 10 librados: watch_flush exit 2024-03-25T19:52:02.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.988+0000 7f0515cc1640 20 client.4669.objecter shutdown clearing up homeless session... 2024-03-25T19:52:02.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.988+0000 7f0515cc1640 10 client.4669.objecter successfully canceled tick 2024-03-25T19:52:02.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.988+0000 7f0515cc1640 1 -- 172.21.15.81:0/2743506765 >> v1:172.21.15.98:6800/3980829775 conn(0x7f04fc06e0f0 legacy=0x7f04fc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:02.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.988+0000 7f0515cc1640 10 monclient: shutdown 2024-03-25T19:52:02.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.988+0000 7f0515cc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:02.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.988+0000 7f0515cc1640 1 -- 172.21.15.81:0/2743506765 >> v1:172.21.15.98:6789/0 conn(0x7f051014be90 legacy=0x7f05101db230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:02.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.989+0000 7f0515cc1640 1 -- 172.21.15.81:0/2743506765 shutdown_connections 2024-03-25T19:52:02.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.989+0000 7f0515cc1640 1 -- 172.21.15.81:0/2743506765 >> 172.21.15.81:0/2743506765 conn(0x7f05100617a0 msgr2=0x7f051013c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:02.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.989+0000 7f0515cc1640 1 -- 172.21.15.81:0/2743506765 shutdown_connections 2024-03-25T19:52:02.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.989+0000 7f0515cc1640 1 -- 172.21.15.81:0/2743506765 wait complete. 2024-03-25T19:52:02.991 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:02.989+0000 7f0515cc1640 1 librados: shutdown 2024-03-25T19:52:03.001 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:08.760 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:10:31.402 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:08.811 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:10:31.453 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:08.853 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:10:31.495 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:08.861 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:10:31.503 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:08.872 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:10:31.514 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:08.879 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:31.514 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:08.883 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:31.518 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:08.909 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:31.544 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:08.915 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:10:31.557 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:09.807 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:09.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.907+0000 7fd13c1f6640 10 monclient: get_monmap_and_config 2024-03-25T19:52:09.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.907+0000 7fd13c1f6640 10 monclient: build_initial_monmap 2024-03-25T19:52:09.907 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.907+0000 7fd13c1f6640 10 monclient: monmap: 2024-03-25T19:52:09.907 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:52:09.907 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:09.907 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:52:09.908237+0000 2024-03-25T19:52:09.907 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:52:09.908237+0000 2024-03-25T19:52:09.907 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:09.907 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:09.907 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:52:09.907 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:52:09.907 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:52:09.908 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:09.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.908+0000 7fd13c1f6640 1 Processor -- start 2024-03-25T19:52:09.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.908+0000 7fd13c1f6640 1 -- start start 2024-03-25T19:52:09.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.908+0000 7fd13c1f6640 10 monclient: init 2024-03-25T19:52:09.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.908+0000 7fd13c1f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:09.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.908+0000 7fd13c1f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:52:09.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.909+0000 7fd13c1f6640 10 monclient(hunting): picked mon.noname-c con 0x7fd1340b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:52:09.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.909+0000 7fd13c1f6640 10 monclient(hunting): picked mon.noname-a con 0x7fd1340b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:52:09.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.909+0000 7fd13c1f6640 10 monclient(hunting): picked mon.noname-b con 0x7fd13414be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:52:09.909 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.909+0000 7fd13c1f6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1340af940 con 0x7fd1340b1730 2024-03-25T19:52:09.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.909+0000 7fd13c1f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1340b0a70 con 0x7fd1340b4d60 2024-03-25T19:52:09.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.909+0000 7fd13c1f6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1340b0d50 con 0x7fd13414be90 2024-03-25T19:52:09.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.909+0000 7fd13c1f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:09.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.909+0000 7fd13c1f6640 10 monclient(hunting): authenticate will time out at 4564.990951s 2024-03-25T19:52:09.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.909+0000 7fd13a793640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fd13414be90 0x7fd13414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:58114/0 (socket says 172.21.15.81:58114) 2024-03-25T19:52:09.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.909+0000 7fd13a793640 1 -- 172.21.15.81:0/3022318867 learned_addr learned my addr 172.21.15.81:0/3022318867 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:52:09.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.977+0000 7fd138f90640 1 -- 172.21.15.81:0/3022318867 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419453937 0 0) 0x7fd1340b0d50 con 0x7fd13414be90 2024-03-25T19:52:09.976 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.977+0000 7fd138f90640 10 monclient(hunting): _negotiate 2024-03-25T19:52:09.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.977+0000 7fd138f90640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:09.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.977+0000 7fd138f90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:09.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.977+0000 7fd138f90640 10 monclient(hunting): my global_id is 4658 2024-03-25T19:52:09.977 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.977+0000 7fd138f90640 1 -- 172.21.15.81:0/3022318867 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd120003520 con 0x7fd13414be90 2024-03-25T19:52:09.978 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.978+0000 7fd138f90640 1 -- 172.21.15.81:0/3022318867 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295942896 0 0) 0x7fd1340af940 con 0x7fd1340b1730 2024-03-25T19:52:09.978 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.978+0000 7fd138f90640 10 monclient(hunting): _negotiate 2024-03-25T19:52:09.978 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.978+0000 7fd138f90640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:09.978 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.978+0000 7fd138f90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:09.978 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.978+0000 7fd138f90640 10 monclient(hunting): my global_id is 4695 2024-03-25T19:52:09.978 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.978+0000 7fd138f90640 1 -- 172.21.15.81:0/3022318867 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1340b0d50 con 0x7fd1340b1730 2024-03-25T19:52:09.987 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.987+0000 7fd138f90640 1 -- 172.21.15.81:0/3022318867 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739408007 0 0) 0x7fd1340b0a70 con 0x7fd1340b4d60 2024-03-25T19:52:09.987 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.987+0000 7fd138f90640 10 monclient(hunting): _negotiate 2024-03-25T19:52:09.987 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.987+0000 7fd138f90640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:09.987 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.987+0000 7fd138f90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:09.987 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.987+0000 7fd138f90640 10 monclient(hunting): my global_id is 4672 2024-03-25T19:52:09.987 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:09.987+0000 7fd138f90640 1 -- 172.21.15.81:0/3022318867 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1340af940 con 0x7fd1340b4d60 2024-03-25T19:52:10.000 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd138f90640 1 -- 172.21.15.81:0/3022318867 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3966038925 0 0) 0x7fd1340af940 con 0x7fd1340b4d60 2024-03-25T19:52:10.000 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd138f90640 1 -- 172.21.15.81:0/3022318867 >> v1:172.21.15.81:6790/0 conn(0x7fd13414be90 legacy=0x7fd13414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:10.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd138f90640 1 -- 172.21.15.81:0/3022318867 >> v1:172.21.15.81:6789/0 conn(0x7fd1340b1730 legacy=0x7fd1340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:10.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd138f90640 10 monclient: _finish_hunting 0 2024-03-25T19:52:10.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd138f90640 1 monclient: found mon.noname-c 2024-03-25T19:52:10.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd138f90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:10.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd138f90640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:52:10.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd138f90640 1 -- 172.21.15.81:0/3022318867 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1340b1030 con 0x7fd1340b4d60 2024-03-25T19:52:10.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd138f90640 10 monclient: _finish_auth 0 2024-03-25T19:52:10.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd138f90640 10 monclient: _check_auth_tickets 2024-03-25T19:52:10.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd138f90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:10.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd138f90640 1 -- 172.21.15.81:0/3022318867 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd1300015b0 con 0x7fd1340b4d60 2024-03-25T19:52:10.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd13c1f6640 5 monclient: authenticate success, global_id 4672 2024-03-25T19:52:10.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd13c1f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:10.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd138f90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:10.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd138f90640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:52:10.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd138f90640 10 monclient: dump: 2024-03-25T19:52:10.002 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:10.002 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:10.002 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:10.002 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:10.002 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:10.002 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:10.002 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:10.002 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:10.002 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:10.002 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:10.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.000+0000 7fd13c1f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:10.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.024+0000 7fd138f90640 1 -- 172.21.15.81:0/3022318867 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd130001710 con 0x7fd1340b4d60 2024-03-25T19:52:10.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.024+0000 7fd138f90640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:10.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.024+0000 7fd138f90640 1 -- 172.21.15.81:0/3022318867 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd130001b20 con 0x7fd1340b4d60 2024-03-25T19:52:10.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.024+0000 7fd138f90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:10.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.024+0000 7fd138f90640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:10.024 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.024+0000 7fd138f90640 10 monclient: dump: 2024-03-25T19:52:10.024 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:10.024 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:10.024 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:10.024 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:10.025 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:10.025 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:10.025 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:10.025 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:10.025 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:10.025 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:10.025 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.024+0000 7fd13c1f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:10.025 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.024+0000 7fd13c1f6640 4 set_mon_vals no callback set 2024-03-25T19:52:10.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.025+0000 7fd13c1f6640 10 monclient: shutdown 2024-03-25T19:52:10.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.025+0000 7fd13c1f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:10.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.025+0000 7fd13c1f6640 1 -- 172.21.15.81:0/3022318867 >> v1:172.21.15.98:6789/0 conn(0x7fd1340b4d60 legacy=0x7fd13413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:10.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.025+0000 7fd13c1f6640 1 -- 172.21.15.81:0/3022318867 shutdown_connections 2024-03-25T19:52:10.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.025+0000 7fd13c1f6640 1 -- 172.21.15.81:0/3022318867 >> 172.21.15.81:0/3022318867 conn(0x7fd1340617a0 msgr2=0x7fd134061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:10.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.025+0000 7fd13c1f6640 1 -- 172.21.15.81:0/3022318867 shutdown_connections 2024-03-25T19:52:10.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.025+0000 7fd13c1f6640 1 -- 172.21.15.81:0/3022318867 wait complete. 2024-03-25T19:52:10.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.026+0000 7fd13c1f6640 10 monclient: build_initial_monmap 2024-03-25T19:52:10.028 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.026+0000 7fd13c1f6640 10 monclient: monmap: 2024-03-25T19:52:10.028 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:52:10.028 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:10.029 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:52:10.029 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:52:10.029 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:10.029 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:52:10.029 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:52:10.029 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:52:10.029 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:52:10.029 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:10.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.026+0000 7fd13c1f6640 1 librados: starting msgr at 2024-03-25T19:52:10.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.026+0000 7fd13c1f6640 1 librados: starting objecter 2024-03-25T19:52:10.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.026+0000 7fd13c1f6640 1 Processor -- start 2024-03-25T19:52:10.029 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.026+0000 7fd13c1f6640 1 -- start start 2024-03-25T19:52:10.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.026+0000 7fd13c1f6640 1 librados: setting wanted keys 2024-03-25T19:52:10.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.026+0000 7fd13c1f6640 1 librados: calling monclient init 2024-03-25T19:52:10.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.026+0000 7fd13c1f6640 10 monclient: init 2024-03-25T19:52:10.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.027+0000 7fd13c1f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:10.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.027+0000 7fd13c1f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:52:10.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.027+0000 7fd13c1f6640 10 monclient(hunting): picked mon.noname-a con 0x7fd1340b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:52:10.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.027+0000 7fd13c1f6640 10 monclient(hunting): picked mon.noname-c con 0x7fd1340b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:52:10.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.027+0000 7fd13c1f6640 10 monclient(hunting): picked mon.noname-b con 0x7fd13414be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:52:10.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.027+0000 7fd13c1f6640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd134141c80 con 0x7fd1340b1730 2024-03-25T19:52:10.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.027+0000 7fd13c1f6640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1341403f0 con 0x7fd13414be90 2024-03-25T19:52:10.030 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.027+0000 7fd13c1f6640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd13413dba0 con 0x7fd1340b4d60 2024-03-25T19:52:10.031 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.027+0000 7fd13c1f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:10.031 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.027+0000 7fd13c1f6640 10 monclient(hunting): authenticate will time out at 4565.109389s 2024-03-25T19:52:10.037 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.038+0000 7fd13a793640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fd13414be90 0x7fd13413e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:34326/0 (socket says 172.21.15.81:34326) 2024-03-25T19:52:10.038 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.038+0000 7fd13a793640 1 -- 172.21.15.81:0/18998381 learned_addr learned my addr 172.21.15.81:0/18998381 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:52:10.038 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.039+0000 7fd12a7fc640 10 client.?.objecter ms_handle_connect 0x7fd13414be90 2024-03-25T19:52:10.039 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.039+0000 7fd12a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:10.049 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.049+0000 7fd12a7fc640 10 client.?.objecter ms_handle_connect 0x7fd1340b1730 2024-03-25T19:52:10.049 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.050+0000 7fd12a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:10.052 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.053+0000 7fd12a7fc640 10 client.?.objecter ms_handle_connect 0x7fd1340b4d60 2024-03-25T19:52:10.052 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.053+0000 7fd12a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:10.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.059+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75880614 0 0) 0x7fd1341403f0 con 0x7fd13414be90 2024-03-25T19:52:10.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.059+0000 7fd12a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:10.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.059+0000 7fd12a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:10.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.059+0000 7fd12a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:10.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.059+0000 7fd12a7fc640 10 monclient(hunting): my global_id is 4675 2024-03-25T19:52:10.059 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.059+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd118004eb0 con 0x7fd13414be90 2024-03-25T19:52:10.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.074+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10563034 0 0) 0x7fd134141c80 con 0x7fd1340b1730 2024-03-25T19:52:10.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.074+0000 7fd12a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:10.074 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.074+0000 7fd12a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:10.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.074+0000 7fd12a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:10.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.074+0000 7fd12a7fc640 10 monclient(hunting): my global_id is 4698 2024-03-25T19:52:10.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.074+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1341403f0 con 0x7fd1340b1730 2024-03-25T19:52:10.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.075+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152664709 0 0) 0x7fd13413dba0 con 0x7fd1340b4d60 2024-03-25T19:52:10.075 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.075+0000 7fd12a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:10.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.075+0000 7fd12a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:10.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.075+0000 7fd12a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:10.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.075+0000 7fd12a7fc640 10 monclient(hunting): my global_id is 4661 2024-03-25T19:52:10.076 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.075+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd134141c80 con 0x7fd1340b4d60 2024-03-25T19:52:10.079 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.079+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1539610450 0 0) 0x7fd118004eb0 con 0x7fd13414be90 2024-03-25T19:52:10.079 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.079+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd13413dba0 con 0x7fd13414be90 2024-03-25T19:52:10.079 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.079+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd12c001b70 con 0x7fd13414be90 2024-03-25T19:52:10.079 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.079+0000 7fd12a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:10.079 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.079+0000 7fd12a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:10.080 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.079+0000 7fd12a7fc640 10 monclient(hunting): dump: 2024-03-25T19:52:10.080 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:10.080 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:10.080 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:10.080 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:10.080 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:10.080 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:10.080 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:10.080 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:10.080 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:10.080 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:10.081 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.079+0000 7fd12a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:10.103 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.103+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1297038299 0 0) 0x7fd1341403f0 con 0x7fd1340b1730 2024-03-25T19:52:10.103 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.103+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd118004eb0 con 0x7fd1340b1730 2024-03-25T19:52:10.103 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.103+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd130001a20 con 0x7fd1340b1730 2024-03-25T19:52:10.103 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.103+0000 7fd12a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:10.103 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.103+0000 7fd12a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:10.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.103+0000 7fd12a7fc640 10 monclient(hunting): dump: 2024-03-25T19:52:10.104 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:10.104 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:10.104 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:10.104 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:10.104 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:10.104 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:10.104 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:10.104 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:10.104 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:10.104 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:10.111 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 59122906 0 0) 0x7fd13413dba0 con 0x7fd13414be90 2024-03-25T19:52:10.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 >> v1:172.21.15.81:6790/0 conn(0x7fd1340b4d60 legacy=0x7fd13413d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:10.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 >> v1:172.21.15.81:6789/0 conn(0x7fd1340b1730 legacy=0x7fd13413c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:10.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd12a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:52:10.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd12a7fc640 1 monclient: found mon.b 2024-03-25T19:52:10.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd12a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:10.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd12a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:10.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13413e7d0 con 0x7fd13414be90 2024-03-25T19:52:10.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd12a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:52:10.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd12a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:52:10.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd12a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:10.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd13c1f6640 5 monclient: authenticate success, global_id 4675 2024-03-25T19:52:10.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd13c1f6640 10 monclient: _renew_subs 2024-03-25T19:52:10.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd13c1f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:10.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd13c1f6640 1 -- 172.21.15.81:0/18998381 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13413ea20 con 0x7fd13414be90 2024-03-25T19:52:10.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd13c1f6640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:10.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd13c1f6640 10 monclient: _renew_subs 2024-03-25T19:52:10.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd13c1f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:10.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd13c1f6640 1 -- 172.21.15.81:0/18998381 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd134141640 con 0x7fd13414be90 2024-03-25T19:52:10.113 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.111+0000 7fd13c1f6640 1 librados: init done 2024-03-25T19:52:10.114 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.112+0000 7fd13c1f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:10.114 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.112+0000 7fd13c1f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:10.114 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.112+0000 7fd13c1f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:10.114 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.112+0000 7fd13c1f6640 1 -- 172.21.15.81:0/18998381 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1340627b0 con 0x7fd13414be90 2024-03-25T19:52:10.412 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.413+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd12c002060 con 0x7fd13414be90 2024-03-25T19:52:10.413 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.413+0000 7fd12a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:10.413 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.413+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fd12c004d00 con 0x7fd13414be90 2024-03-25T19:52:10.413 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.413+0000 7fd12a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:10.413 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.413+0000 7fd12a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:10.413 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.413+0000 7fd12a7fc640 10 monclient: dump: 2024-03-25T19:52:10.413 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:10.413 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:10.413 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:10.413 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:10.413 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:10.413 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:10.413 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:10.413 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:10.414 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:10.414 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:10.414 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.413+0000 7fd12bfff640 4 set_mon_vals no callback set 2024-03-25T19:52:10.414 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.414+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fd12c01ad10 con 0x7fd13414be90 2024-03-25T19:52:10.414 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.414+0000 7fd139f92640 10 monclient: get_auth_request con 0x7fd11806ded0 auth_method 0 2024-03-25T19:52:10.414 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.415+0000 7fd12a7fc640 10 client.4675.objecter ms_handle_connect 0x7fd11806ded0 2024-03-25T19:52:10.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.418+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3447814714 0 0) 0x7fd12c054a40 con 0x7fd13414be90 2024-03-25T19:52:10.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.418+0000 7fd12a7fc640 10 client.4675.objecter ms_dispatch 0x7fd134000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:10.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.418+0000 7fd12a7fc640 3 client.4675.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:10.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.418+0000 7fd12a7fc640 3 client.4675.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:10.418 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.418+0000 7fd12a7fc640 20 client.4675.objecter dump_active .. 0 homeless 2024-03-25T19:52:10.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.419+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd12c050380 con 0x7fd13414be90 2024-03-25T19:52:10.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.419+0000 7fd12a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:10.419 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.419+0000 7fd12a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:10.668 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.669+0000 7fd13c1f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:10.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.669+0000 7fd13c1f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:10.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.669+0000 7fd13c1f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:10.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.669+0000 7fd13c1f6640 1 -- 172.21.15.81:0/18998381 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd134062070 con 0x7fd13414be90 2024-03-25T19:52:10.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.724+0000 7fd12a7fc640 1 -- 172.21.15.81:0/18998381 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3235219182) 0x7fd12c052480 con 0x7fd13414be90 2024-03-25T19:52:10.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.724+0000 7fd12a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:10.724 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.724+0000 7fd12a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:10.724 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:52:10.724 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":11,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:52:03.690842+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:52:03.690842+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:10.726 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.726+0000 7fd13c1f6640 10 librados: watch_flush enter 2024-03-25T19:52:10.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.726+0000 7fd13c1f6640 10 librados: watch_flush exit 2024-03-25T19:52:10.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.726+0000 7fd13c1f6640 20 client.4675.objecter shutdown clearing up homeless session... 2024-03-25T19:52:10.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.726+0000 7fd13c1f6640 10 client.4675.objecter successfully canceled tick 2024-03-25T19:52:10.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.726+0000 7fd13c1f6640 1 -- 172.21.15.81:0/18998381 >> v1:172.21.15.98:6800/3980829775 conn(0x7fd11806ded0 legacy=0x7fd118070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:10.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.726+0000 7fd13c1f6640 10 monclient: shutdown 2024-03-25T19:52:10.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.726+0000 7fd13c1f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:10.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.726+0000 7fd13c1f6640 1 -- 172.21.15.81:0/18998381 >> v1:172.21.15.98:6789/0 conn(0x7fd13414be90 legacy=0x7fd13413e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:10.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.726+0000 7fd13c1f6640 1 -- 172.21.15.81:0/18998381 shutdown_connections 2024-03-25T19:52:10.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.726+0000 7fd13c1f6640 1 -- 172.21.15.81:0/18998381 >> 172.21.15.81:0/18998381 conn(0x7fd1340617a0 msgr2=0x7fd134143900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:10.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.726+0000 7fd13c1f6640 1 -- 172.21.15.81:0/18998381 shutdown_connections 2024-03-25T19:52:10.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.726+0000 7fd13c1f6640 1 -- 172.21.15.81:0/18998381 wait complete. 2024-03-25T19:52:10.727 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:10.727+0000 7fd13c1f6640 1 librados: shutdown 2024-03-25T19:52:10.737 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:17.543 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:17.642 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.642+0000 7fef5cae4640 10 monclient: get_monmap_and_config 2024-03-25T19:52:17.642 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.642+0000 7fef5cae4640 10 monclient: build_initial_monmap 2024-03-25T19:52:17.642 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.642+0000 7fef5cae4640 10 monclient: monmap: 2024-03-25T19:52:17.642 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:52:17.642 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:17.642 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:52:17.643297+0000 2024-03-25T19:52:17.642 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:52:17.643297+0000 2024-03-25T19:52:17.642 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:17.643 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:17.643 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:52:17.643 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:52:17.643 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:52:17.643 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:17.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.643+0000 7fef5cae4640 1 Processor -- start 2024-03-25T19:52:17.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.643+0000 7fef5cae4640 1 -- start start 2024-03-25T19:52:17.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.643+0000 7fef5cae4640 10 monclient: init 2024-03-25T19:52:17.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.643+0000 7fef5cae4640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:17.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.643+0000 7fef5cae4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:52:17.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.643+0000 7fef5cae4640 10 monclient(hunting): picked mon.noname-c con 0x7fef580a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:52:17.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.643+0000 7fef5cae4640 10 monclient(hunting): picked mon.noname-a con 0x7fef5813f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:52:17.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.643+0000 7fef5cae4640 10 monclient(hunting): picked mon.noname-b con 0x7fef581430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:52:17.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.643+0000 7fef5cae4640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef58142520 con 0x7fef5813f710 2024-03-25T19:52:17.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.643+0000 7fef5cae4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef580b28a0 con 0x7fef580a98f0 2024-03-25T19:52:17.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.643+0000 7fef5cae4640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef580b2a30 con 0x7fef581430c0 2024-03-25T19:52:17.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.643+0000 7fef5cae4640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:17.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.643+0000 7fef5cae4640 10 monclient(hunting): authenticate will time out at 4572.725719s 2024-03-25T19:52:17.656 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.657+0000 7fef4dd9b640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fef5813f710 0x7fef581429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:44966/0 (socket says 172.21.15.81:44966) 2024-03-25T19:52:17.657 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.657+0000 7fef4dd9b640 1 -- 172.21.15.81:0/835380546 learned_addr learned my addr 172.21.15.81:0/835380546 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:52:17.677 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.678+0000 7fef55d9b640 1 -- 172.21.15.81:0/835380546 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262861011 0 0) 0x7fef580b28a0 con 0x7fef580a98f0 2024-03-25T19:52:17.677 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.678+0000 7fef55d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:52:17.677 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.678+0000 7fef55d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:17.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.678+0000 7fef55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:17.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.678+0000 7fef55d9b640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:52:17.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.678+0000 7fef55d9b640 1 -- 172.21.15.81:0/835380546 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef44003520 con 0x7fef580a98f0 2024-03-25T19:52:17.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.678+0000 7fef55d9b640 1 -- 172.21.15.81:0/835380546 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324246216 0 0) 0x7fef58142520 con 0x7fef5813f710 2024-03-25T19:52:17.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.678+0000 7fef55d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:52:17.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.678+0000 7fef55d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:17.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.678+0000 7fef55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:17.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.678+0000 7fef55d9b640 10 monclient(hunting): my global_id is 4701 2024-03-25T19:52:17.678 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.678+0000 7fef55d9b640 1 -- 172.21.15.81:0/835380546 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef580b28a0 con 0x7fef5813f710 2024-03-25T19:52:17.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.700+0000 7fef55d9b640 1 -- 172.21.15.81:0/835380546 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 642761899 0 0) 0x7fef580b28a0 con 0x7fef5813f710 2024-03-25T19:52:17.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.700+0000 7fef55d9b640 1 -- 172.21.15.81:0/835380546 >> v1:172.21.15.81:6790/0 conn(0x7fef581430c0 legacy=0x7fef5814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:17.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.700+0000 7fef55d9b640 1 -- 172.21.15.81:0/835380546 >> v1:172.21.15.98:6789/0 conn(0x7fef580a98f0 legacy=0x7fef580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:17.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.700+0000 7fef55d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:52:17.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.700+0000 7fef55d9b640 1 monclient: found mon.noname-a 2024-03-25T19:52:17.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.700+0000 7fef55d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:17.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.700+0000 7fef55d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:52:17.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.700+0000 7fef55d9b640 1 -- 172.21.15.81:0/835380546 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef580b44c0 con 0x7fef5813f710 2024-03-25T19:52:17.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.700+0000 7fef55d9b640 10 monclient: _finish_auth 0 2024-03-25T19:52:17.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.700+0000 7fef55d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:52:17.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.700+0000 7fef55d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:17.700 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.700+0000 7fef55d9b640 1 -- 172.21.15.81:0/835380546 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fef40001670 con 0x7fef5813f710 2024-03-25T19:52:17.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.700+0000 7fef55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:17.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.700+0000 7fef55d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:17.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.700+0000 7fef55d9b640 10 monclient: dump: 2024-03-25T19:52:17.701 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:17.701 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:17.701 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:17.701 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:17.701 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:17.701 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:17.701 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:17.701 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:17.701 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:17.701 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:17.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.700+0000 7fef5cae4640 5 monclient: authenticate success, global_id 4701 2024-03-25T19:52:17.701 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.700+0000 7fef5cae4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:17.712 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.712+0000 7fef55d9b640 1 -- 172.21.15.81:0/835380546 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef400017d0 con 0x7fef5813f710 2024-03-25T19:52:17.712 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.712+0000 7fef55d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:17.712 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.712+0000 7fef55d9b640 1 -- 172.21.15.81:0/835380546 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fef40001ba0 con 0x7fef5813f710 2024-03-25T19:52:17.712 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.712+0000 7fef55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:17.712 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.712+0000 7fef55d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:17.712 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.712+0000 7fef55d9b640 10 monclient: dump: 2024-03-25T19:52:17.712 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:17.712 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:17.713 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:17.713 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:17.713 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:17.713 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:17.713 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:17.713 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:17.713 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:17.713 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:17.713 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.712+0000 7fef5cae4640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:17.713 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.712+0000 7fef5cae4640 4 set_mon_vals no callback set 2024-03-25T19:52:17.715 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.713+0000 7fef5cae4640 10 monclient: shutdown 2024-03-25T19:52:17.715 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.713+0000 7fef5cae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:17.715 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.713+0000 7fef5cae4640 1 -- 172.21.15.81:0/835380546 >> v1:172.21.15.81:6789/0 conn(0x7fef5813f710 legacy=0x7fef581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:17.715 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.713+0000 7fef5cae4640 1 -- 172.21.15.81:0/835380546 shutdown_connections 2024-03-25T19:52:17.715 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.713+0000 7fef5cae4640 1 -- 172.21.15.81:0/835380546 >> 172.21.15.81:0/835380546 conn(0x7fef580b7ea0 msgr2=0x7fef580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:17.715 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.713+0000 7fef5cae4640 1 -- 172.21.15.81:0/835380546 shutdown_connections 2024-03-25T19:52:17.715 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.713+0000 7fef5cae4640 1 -- 172.21.15.81:0/835380546 wait complete. 2024-03-25T19:52:17.715 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.714+0000 7fef5cae4640 10 monclient: build_initial_monmap 2024-03-25T19:52:17.715 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.714+0000 7fef5cae4640 10 monclient: monmap: 2024-03-25T19:52:17.715 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:52:17.715 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:17.715 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:52:17.715 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:52:17.716 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:17.716 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:52:17.716 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:52:17.716 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:52:17.716 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:52:17.716 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:17.716 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.714+0000 7fef5cae4640 1 librados: starting msgr at 2024-03-25T19:52:17.716 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.714+0000 7fef5cae4640 1 librados: starting objecter 2024-03-25T19:52:17.716 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.715+0000 7fef5cae4640 1 Processor -- start 2024-03-25T19:52:17.716 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.715+0000 7fef5cae4640 1 -- start start 2024-03-25T19:52:17.716 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.715+0000 7fef5cae4640 1 librados: setting wanted keys 2024-03-25T19:52:17.716 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.715+0000 7fef5cae4640 1 librados: calling monclient init 2024-03-25T19:52:17.716 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.715+0000 7fef5cae4640 10 monclient: init 2024-03-25T19:52:17.717 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.715+0000 7fef5cae4640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:17.717 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.715+0000 7fef5cae4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:52:17.717 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.715+0000 7fef5cae4640 10 monclient(hunting): picked mon.noname-b con 0x7fef580a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:52:17.717 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.716+0000 7fef5cae4640 10 monclient(hunting): picked mon.noname-c con 0x7fef5813f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:52:17.717 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.716+0000 7fef5cae4640 10 monclient(hunting): picked mon.noname-a con 0x7fef581430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:52:17.717 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.716+0000 7fef5cae4640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef581d8440 con 0x7fef581430c0 2024-03-25T19:52:17.717 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.716+0000 7fef5cae4640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef581d7810 con 0x7fef580a98f0 2024-03-25T19:52:17.718 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.716+0000 7fef5cae4640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef581d50a0 con 0x7fef5813f710 2024-03-25T19:52:17.718 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.716+0000 7fef5cae4640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:17.718 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.716+0000 7fef5cae4640 10 monclient(hunting): authenticate will time out at 4572.798020s 2024-03-25T19:52:17.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.723+0000 7fef5659c640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fef580a98f0 0x7fef581d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:51234/0 (socket says 172.21.15.81:51234) 2024-03-25T19:52:17.722 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.723+0000 7fef5659c640 1 -- 172.21.15.81:0/316801756 learned_addr learned my addr 172.21.15.81:0/316801756 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:52:17.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.723+0000 7fef4f7fe640 10 client.?.objecter ms_handle_connect 0x7fef580a98f0 2024-03-25T19:52:17.723 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.723+0000 7fef4f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:17.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.729+0000 7fef4f7fe640 10 client.?.objecter ms_handle_connect 0x7fef5813f710 2024-03-25T19:52:17.729 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.729+0000 7fef4f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:17.731 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.731+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421411629 0 0) 0x7fef581d7810 con 0x7fef580a98f0 2024-03-25T19:52:17.731 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.731+0000 7fef4f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:52:17.731 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.731+0000 7fef4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:17.731 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.731+0000 7fef4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:17.731 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.731+0000 7fef4f7fe640 10 monclient(hunting): my global_id is 4681 2024-03-25T19:52:17.731 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.731+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef38004eb0 con 0x7fef580a98f0 2024-03-25T19:52:17.734 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.735+0000 7fef4f7fe640 10 client.?.objecter ms_handle_connect 0x7fef581430c0 2024-03-25T19:52:17.735 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.735+0000 7fef4f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:17.743 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.743+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452129552 0 0) 0x7fef581d50a0 con 0x7fef5813f710 2024-03-25T19:52:17.743 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.743+0000 7fef4f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:52:17.743 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.743+0000 7fef4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:17.743 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.743+0000 7fef4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:17.743 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.743+0000 7fef4f7fe640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:52:17.743 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.743+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef581d7810 con 0x7fef5813f710 2024-03-25T19:52:17.746 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.746+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78120086 0 0) 0x7fef581d8440 con 0x7fef581430c0 2024-03-25T19:52:17.746 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.746+0000 7fef4f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:52:17.746 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.746+0000 7fef4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:17.746 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.746+0000 7fef4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:17.746 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.746+0000 7fef4f7fe640 10 monclient(hunting): my global_id is 4704 2024-03-25T19:52:17.746 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.746+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef581d50a0 con 0x7fef581430c0 2024-03-25T19:52:17.752 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.752+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2207647994 0 0) 0x7fef38004eb0 con 0x7fef580a98f0 2024-03-25T19:52:17.752 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.752+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef581d8440 con 0x7fef580a98f0 2024-03-25T19:52:17.752 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.752+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fef48001a10 con 0x7fef580a98f0 2024-03-25T19:52:17.752 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.752+0000 7fef4f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:17.752 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.752+0000 7fef4f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:17.752 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.752+0000 7fef4f7fe640 10 monclient(hunting): dump: 2024-03-25T19:52:17.753 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:17.753 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:17.753 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:17.753 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:17.753 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:17.753 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:17.753 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:17.753 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:17.753 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:17.753 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:17.753 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.752+0000 7fef4f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:17.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.763+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3582439025 0 0) 0x7fef581d8440 con 0x7fef580a98f0 2024-03-25T19:52:17.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 >> v1:172.21.15.81:6790/0 conn(0x7fef5813f710 legacy=0x7fef581d4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:17.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 >> v1:172.21.15.81:6789/0 conn(0x7fef581430c0 legacy=0x7fef581384c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:17.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef4f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:52:17.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef4f7fe640 1 monclient: found mon.b 2024-03-25T19:52:17.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:17.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef4f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:17.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef581d5230 con 0x7fef580a98f0 2024-03-25T19:52:17.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef4f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:52:17.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef4f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:52:17.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:17.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef5cae4640 5 monclient: authenticate success, global_id 4681 2024-03-25T19:52:17.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef5cae4640 10 monclient: _renew_subs 2024-03-25T19:52:17.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef5cae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:17.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef5cae4640 1 -- 172.21.15.81:0/316801756 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef58137f70 con 0x7fef580a98f0 2024-03-25T19:52:17.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef5cae4640 10 client.4681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:17.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef5cae4640 10 monclient: _renew_subs 2024-03-25T19:52:17.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef5cae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:17.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef5cae4640 1 -- 172.21.15.81:0/316801756 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef581d41e0 con 0x7fef580a98f0 2024-03-25T19:52:17.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.764+0000 7fef5cae4640 1 librados: init done 2024-03-25T19:52:17.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.765+0000 7fef5cae4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:17.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.765+0000 7fef5cae4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:17.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.765+0000 7fef5cae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:17.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:17.765+0000 7fef5cae4640 1 -- 172.21.15.81:0/316801756 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef580abda0 con 0x7fef580a98f0 2024-03-25T19:52:18.044 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.045+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef48001c50 con 0x7fef580a98f0 2024-03-25T19:52:18.045 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.045+0000 7fef4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:18.045 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.045+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fef48003a60 con 0x7fef580a98f0 2024-03-25T19:52:18.045 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.045+0000 7fef4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:18.045 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.045+0000 7fef4f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:18.045 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.045+0000 7fef4f7fe640 10 monclient: dump: 2024-03-25T19:52:18.045 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:18.045 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:18.045 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:18.045 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:18.045 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:18.045 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:18.045 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:18.046 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:18.046 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:18.046 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:18.046 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.045+0000 7fef5559a640 4 set_mon_vals no callback set 2024-03-25T19:52:18.046 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.046+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fef48003c80 con 0x7fef580a98f0 2024-03-25T19:52:18.046 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.047+0000 7fef4dd9b640 10 monclient: get_auth_request con 0x7fef3806ddf0 auth_method 0 2024-03-25T19:52:18.047 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.047+0000 7fef4f7fe640 10 client.4681.objecter ms_handle_connect 0x7fef3806ddf0 2024-03-25T19:52:18.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.050+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3447814714 0 0) 0x7fef48054850 con 0x7fef580a98f0 2024-03-25T19:52:18.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.050+0000 7fef4f7fe640 10 client.4681.objecter ms_dispatch 0x7fef58000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:18.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.051+0000 7fef4f7fe640 3 client.4681.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:18.050 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.051+0000 7fef4f7fe640 3 client.4681.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:18.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.051+0000 7fef4f7fe640 20 client.4681.objecter dump_active .. 0 homeless 2024-03-25T19:52:18.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.051+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef48050360 con 0x7fef580a98f0 2024-03-25T19:52:18.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.051+0000 7fef4f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:18.051 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.051+0000 7fef4f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:18.295 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.295+0000 7fef5cae4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:18.295 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.295+0000 7fef5cae4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:18.295 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.295+0000 7fef5cae4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:18.295 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.295+0000 7fef5cae4640 1 -- 172.21.15.81:0/316801756 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef581d79a0 con 0x7fef580a98f0 2024-03-25T19:52:18.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.318+0000 7fef4f7fe640 1 -- 172.21.15.81:0/316801756 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3235219182) 0x7fef48052290 con 0x7fef580a98f0 2024-03-25T19:52:18.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.318+0000 7fef4f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:18.317 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.318+0000 7fef4f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:18.318 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:52:18.318 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":11,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:52:03.690842+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:52:03.690842+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:18.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.320+0000 7fef5cae4640 10 librados: watch_flush enter 2024-03-25T19:52:18.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.320+0000 7fef5cae4640 10 librados: watch_flush exit 2024-03-25T19:52:18.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.320+0000 7fef5cae4640 20 client.4681.objecter shutdown clearing up homeless session... 2024-03-25T19:52:18.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.320+0000 7fef5cae4640 10 client.4681.objecter successfully canceled tick 2024-03-25T19:52:18.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.320+0000 7fef5cae4640 1 -- 172.21.15.81:0/316801756 >> v1:172.21.15.98:6800/3980829775 conn(0x7fef3806ddf0 legacy=0x7fef380702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:18.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.320+0000 7fef5cae4640 10 monclient: shutdown 2024-03-25T19:52:18.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.320+0000 7fef5cae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:18.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.320+0000 7fef5cae4640 1 -- 172.21.15.81:0/316801756 >> v1:172.21.15.98:6789/0 conn(0x7fef580a98f0 legacy=0x7fef581d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:18.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.320+0000 7fef5cae4640 1 -- 172.21.15.81:0/316801756 shutdown_connections 2024-03-25T19:52:18.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.320+0000 7fef5cae4640 1 -- 172.21.15.81:0/316801756 >> 172.21.15.81:0/316801756 conn(0x7fef580b7ea0 msgr2=0x7fef58074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:18.321 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.320+0000 7fef5cae4640 1 -- 172.21.15.81:0/316801756 shutdown_connections 2024-03-25T19:52:18.322 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.320+0000 7fef5cae4640 1 -- 172.21.15.81:0/316801756 wait complete. 2024-03-25T19:52:18.322 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:18.320+0000 7fef5cae4640 1 librados: shutdown 2024-03-25T19:52:18.333 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:25.139 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:25.255 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.255+0000 7f57e98a8640 10 monclient: get_monmap_and_config 2024-03-25T19:52:25.255 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.255+0000 7f57e98a8640 10 monclient: build_initial_monmap 2024-03-25T19:52:25.255 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.255+0000 7f57e98a8640 10 monclient: monmap: 2024-03-25T19:52:25.255 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:52:25.255 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:25.255 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:52:25.256669+0000 2024-03-25T19:52:25.256 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:52:25.256669+0000 2024-03-25T19:52:25.256 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:25.256 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:25.256 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:52:25.256 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:52:25.256 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:52:25.256 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:25.257 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.256+0000 7f57e98a8640 1 Processor -- start 2024-03-25T19:52:25.257 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.256+0000 7f57e98a8640 1 -- start start 2024-03-25T19:52:25.257 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.256+0000 7f57e98a8640 10 monclient: init 2024-03-25T19:52:25.257 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.257+0000 7f57e98a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:25.257 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.257+0000 7f57e98a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:52:25.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.257+0000 7f57e98a8640 10 monclient(hunting): picked mon.noname-c con 0x7f57e40a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:52:25.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.257+0000 7f57e98a8640 10 monclient(hunting): picked mon.noname-b con 0x7f57e413f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:52:25.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.257+0000 7f57e98a8640 10 monclient(hunting): picked mon.noname-a con 0x7f57e41430c0 addr v1:172.21.15.81:6789/0 2024-03-25T19:52:25.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.257+0000 7f57e98a8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e4142520 con 0x7f57e41430c0 2024-03-25T19:52:25.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.257+0000 7f57e98a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e40b28a0 con 0x7f57e40a98f0 2024-03-25T19:52:25.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.257+0000 7f57e98a8640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e40b2a30 con 0x7f57e413f710 2024-03-25T19:52:25.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.257+0000 7f57e98a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:25.258 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.257+0000 7f57e98a8640 10 monclient(hunting): authenticate will time out at 4580.339118s 2024-03-25T19:52:25.274 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.275+0000 7f57e27fc640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f57e413f710 0x7f57e41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:51142/0 (socket says 172.21.15.81:51142) 2024-03-25T19:52:25.274 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.275+0000 7f57e27fc640 1 -- 172.21.15.81:0/1166355808 learned_addr learned my addr 172.21.15.81:0/1166355808 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:52:25.296 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.296+0000 7f57e1ffb640 1 -- 172.21.15.81:0/1166355808 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323194224 0 0) 0x7f57e40b28a0 con 0x7f57e40a98f0 2024-03-25T19:52:25.296 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.297+0000 7f57e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:52:25.296 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.297+0000 7f57e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:25.296 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.297+0000 7f57e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:25.296 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.297+0000 7f57e1ffb640 10 monclient(hunting): my global_id is 4684 2024-03-25T19:52:25.297 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.297+0000 7f57e1ffb640 1 -- 172.21.15.81:0/1166355808 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57cc003520 con 0x7f57e40a98f0 2024-03-25T19:52:25.297 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.297+0000 7f57e1ffb640 1 -- 172.21.15.81:0/1166355808 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1129114484 0 0) 0x7f57e40b2a30 con 0x7f57e413f710 2024-03-25T19:52:25.297 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.297+0000 7f57e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:52:25.297 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.297+0000 7f57e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:25.297 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.297+0000 7f57e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:25.297 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.297+0000 7f57e1ffb640 10 monclient(hunting): my global_id is 4670 2024-03-25T19:52:25.297 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.297+0000 7f57e1ffb640 1 -- 172.21.15.81:0/1166355808 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e40b28a0 con 0x7f57e413f710 2024-03-25T19:52:25.306 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.307+0000 7f57e1ffb640 1 -- 172.21.15.81:0/1166355808 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673238801 0 0) 0x7f57e4142520 con 0x7f57e41430c0 2024-03-25T19:52:25.307 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.307+0000 7f57e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:52:25.307 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.307+0000 7f57e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:25.307 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.307+0000 7f57e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:25.307 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.307+0000 7f57e1ffb640 10 monclient(hunting): my global_id is 4707 2024-03-25T19:52:25.307 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.307+0000 7f57e1ffb640 1 -- 172.21.15.81:0/1166355808 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e40b2a30 con 0x7f57e41430c0 2024-03-25T19:52:25.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e1ffb640 1 -- 172.21.15.81:0/1166355808 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2061472536 0 0) 0x7f57e40b28a0 con 0x7f57e413f710 2024-03-25T19:52:25.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e1ffb640 1 -- 172.21.15.81:0/1166355808 >> v1:172.21.15.98:6789/0 conn(0x7f57e40a98f0 legacy=0x7f57e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:25.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e1ffb640 1 -- 172.21.15.81:0/1166355808 >> v1:172.21.15.81:6789/0 conn(0x7f57e41430c0 legacy=0x7f57e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:25.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:52:25.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e1ffb640 1 monclient: found mon.noname-b 2024-03-25T19:52:25.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:25.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:52:25.329 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e1ffb640 1 -- 172.21.15.81:0/1166355808 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e40b44c0 con 0x7f57e413f710 2024-03-25T19:52:25.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:52:25.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:52:25.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:25.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e98a8640 5 monclient: authenticate success, global_id 4670 2024-03-25T19:52:25.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e98a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:25.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e1ffb640 1 -- 172.21.15.81:0/1166355808 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f57d8001670 con 0x7f57e413f710 2024-03-25T19:52:25.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:25.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:25.330 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e1ffb640 10 monclient: dump: 2024-03-25T19:52:25.330 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:25.330 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:25.330 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:25.330 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:25.330 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:25.331 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:25.331 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:25.331 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:25.331 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:25.331 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:25.331 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.329+0000 7f57e98a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:25.357 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.357+0000 7f57e1ffb640 1 -- 172.21.15.81:0/1166355808 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57d8001850 con 0x7f57e413f710 2024-03-25T19:52:25.357 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.357+0000 7f57e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:25.357 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.357+0000 7f57e1ffb640 1 -- 172.21.15.81:0/1166355808 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f57d8001be0 con 0x7f57e413f710 2024-03-25T19:52:25.357 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.357+0000 7f57e98a8640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:25.358 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.357+0000 7f57e98a8640 4 set_mon_vals no callback set 2024-03-25T19:52:25.358 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.357+0000 7f57e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:25.358 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.357+0000 7f57e1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:52:25.358 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.357+0000 7f57e1ffb640 10 monclient: dump: 2024-03-25T19:52:25.358 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:25.358 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:25.359 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:25.359 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:25.359 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:25.359 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:25.359 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:25.360 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:25.360 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:25.360 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:25.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.358+0000 7f57e98a8640 10 monclient: shutdown 2024-03-25T19:52:25.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.358+0000 7f57e98a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:25.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.358+0000 7f57e98a8640 1 -- 172.21.15.81:0/1166355808 >> v1:172.21.15.81:6790/0 conn(0x7f57e413f710 legacy=0x7f57e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:25.363 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.358+0000 7f57e98a8640 1 -- 172.21.15.81:0/1166355808 shutdown_connections 2024-03-25T19:52:25.364 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.358+0000 7f57e98a8640 1 -- 172.21.15.81:0/1166355808 >> 172.21.15.81:0/1166355808 conn(0x7f57e40b7ea0 msgr2=0x7f57e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:25.364 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.358+0000 7f57e98a8640 1 -- 172.21.15.81:0/1166355808 shutdown_connections 2024-03-25T19:52:25.364 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.358+0000 7f57e98a8640 1 -- 172.21.15.81:0/1166355808 wait complete. 2024-03-25T19:52:25.364 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.359+0000 7f57e98a8640 10 monclient: build_initial_monmap 2024-03-25T19:52:25.364 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.359+0000 7f57e98a8640 10 monclient: monmap: 2024-03-25T19:52:25.364 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:52:25.364 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:25.365 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:52:25.365 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:52:25.365 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:25.365 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:52:25.365 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:52:25.365 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:52:25.365 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:52:25.365 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:25.366 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.359+0000 7f57e98a8640 1 librados: starting msgr at 2024-03-25T19:52:25.366 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.359+0000 7f57e98a8640 1 librados: starting objecter 2024-03-25T19:52:25.366 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.359+0000 7f57e98a8640 1 Processor -- start 2024-03-25T19:52:25.366 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.360+0000 7f57e98a8640 1 -- start start 2024-03-25T19:52:25.366 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.360+0000 7f57e98a8640 1 librados: setting wanted keys 2024-03-25T19:52:25.366 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.360+0000 7f57e98a8640 1 librados: calling monclient init 2024-03-25T19:52:25.366 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.360+0000 7f57e98a8640 10 monclient: init 2024-03-25T19:52:25.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.360+0000 7f57e98a8640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:25.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.360+0000 7f57e98a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:52:25.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.360+0000 7f57e98a8640 10 monclient(hunting): picked mon.noname-b con 0x7f57e40a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:52:25.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.360+0000 7f57e98a8640 10 monclient(hunting): picked mon.noname-a con 0x7f57e413f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:52:25.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.360+0000 7f57e98a8640 10 monclient(hunting): picked mon.noname-c con 0x7f57e41430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:52:25.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.360+0000 7f57e98a8640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e4138700 con 0x7f57e413f710 2024-03-25T19:52:25.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.360+0000 7f57e98a8640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e4138b90 con 0x7f57e40a98f0 2024-03-25T19:52:25.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.360+0000 7f57e98a8640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e40b16d0 con 0x7f57e41430c0 2024-03-25T19:52:25.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.360+0000 7f57e98a8640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:25.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.360+0000 7f57e98a8640 10 monclient(hunting): authenticate will time out at 4580.442818s 2024-03-25T19:52:25.377 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.377+0000 7f57e27fc640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f57e413f710 0x7f57e41da880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:52978/0 (socket says 172.21.15.81:52978) 2024-03-25T19:52:25.377 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.377+0000 7f57e27fc640 1 -- 172.21.15.81:0/1211731354 learned_addr learned my addr 172.21.15.81:0/1211731354 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:52:25.378 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.379+0000 7f57cbfff640 10 client.?.objecter ms_handle_connect 0x7f57e413f710 2024-03-25T19:52:25.378 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.379+0000 7f57cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:25.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.383+0000 7f57cbfff640 10 client.?.objecter ms_handle_connect 0x7f57e40a98f0 2024-03-25T19:52:25.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.383+0000 7f57cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:25.390 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.391+0000 7f57cbfff640 1 -- 172.21.15.81:0/1211731354 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529035644 0 0) 0x7f57e4138b90 con 0x7f57e40a98f0 2024-03-25T19:52:25.391 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.391+0000 7f57cbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:25.391 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.391+0000 7f57cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:25.391 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.391+0000 7f57cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:25.391 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.391+0000 7f57cbfff640 10 monclient(hunting): my global_id is 4687 2024-03-25T19:52:25.391 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.391+0000 7f57cbfff640 1 -- 172.21.15.81:0/1211731354 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d0004eb0 con 0x7f57e40a98f0 2024-03-25T19:52:25.394 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.394+0000 7f57cbfff640 1 -- 172.21.15.81:0/1211731354 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354683406 0 0) 0x7f57e4138700 con 0x7f57e413f710 2024-03-25T19:52:25.394 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.394+0000 7f57cbfff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:25.394 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.394+0000 7f57cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:25.394 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.394+0000 7f57cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:25.394 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.394+0000 7f57cbfff640 10 monclient(hunting): my global_id is 4710 2024-03-25T19:52:25.395 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.394+0000 7f57cbfff640 1 -- 172.21.15.81:0/1211731354 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e4138b90 con 0x7f57e413f710 2024-03-25T19:52:25.395 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.395+0000 7f57cbfff640 10 client.?.objecter ms_handle_connect 0x7f57e41430c0 2024-03-25T19:52:25.395 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.395+0000 7f57cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:25.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.397+0000 7f57e37fe640 1 -- 172.21.15.81:0/1211731354 >> v1:172.21.15.81:6790/0 conn(0x7f57e41430c0 legacy=0x7f57e41e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T19:52:25.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.397+0000 7f57e37fe640 1 -- 172.21.15.81:0/1211731354 >> v1:172.21.15.81:6790/0 conn(0x7f57e41430c0 legacy=0x7f57e41e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:52:25.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.397+0000 7f57e37fe640 1 --1- 172.21.15.81:0/1211731354 >> v1:172.21.15.81:6790/0 conn(0x7f57e41430c0 0x7f57e41e1dc0 :-1 s=OPENED pgs=79945 cs=1 l=1).handle_message read tag failed 2024-03-25T19:52:25.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.397+0000 7f57e37fe640 1 --1- 172.21.15.81:0/1211731354 >> v1:172.21.15.81:6790/0 conn(0x7f57e41430c0 0x7f57e41e1dc0 :-1 s=OPENED pgs=79945 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:52:25.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.397+0000 7f57cbfff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.81:6790/0 2024-03-25T19:52:25.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.402+0000 7f57cbfff640 1 -- 172.21.15.81:0/1211731354 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 315248296 0 0) 0x7f57d0004eb0 con 0x7f57e40a98f0 2024-03-25T19:52:25.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.402+0000 7f57cbfff640 1 -- 172.21.15.81:0/1211731354 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57e4138700 con 0x7f57e40a98f0 2024-03-25T19:52:25.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.402+0000 7f57cbfff640 1 -- 172.21.15.81:0/1211731354 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f57d4001950 con 0x7f57e40a98f0 2024-03-25T19:52:25.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.402+0000 7f57cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:25.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.402+0000 7f57cbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:25.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.402+0000 7f57cbfff640 10 monclient(hunting): dump: 2024-03-25T19:52:25.403 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:25.403 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:25.403 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:25.403 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:25.403 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:25.403 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:25.403 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:25.403 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:25.403 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:25.403 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:25.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.402+0000 7f57cbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:25.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57cbfff640 1 -- 172.21.15.81:0/1211731354 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1362376403 0 0) 0x7f57e4138700 con 0x7f57e40a98f0 2024-03-25T19:52:25.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57cbfff640 1 -- 172.21.15.81:0/1211731354 >> v1:172.21.15.81:6790/0 conn(0x7f57e41430c0 legacy=0x7f57e41e1dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:52:25.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57cbfff640 1 -- 172.21.15.81:0/1211731354 >> v1:172.21.15.81:6789/0 conn(0x7f57e413f710 legacy=0x7f57e41da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:25.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57cbfff640 10 monclient: _finish_hunting 0 2024-03-25T19:52:25.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57cbfff640 1 monclient: found mon.b 2024-03-25T19:52:25.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:25.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57cbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:25.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57cbfff640 1 -- 172.21.15.81:0/1211731354 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e4138050 con 0x7f57e40a98f0 2024-03-25T19:52:25.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57cbfff640 10 monclient: _finish_auth 0 2024-03-25T19:52:25.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57cbfff640 10 monclient: _check_auth_tickets 2024-03-25T19:52:25.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:25.408 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57e98a8640 5 monclient: authenticate success, global_id 4687 2024-03-25T19:52:25.409 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57e98a8640 10 monclient: _renew_subs 2024-03-25T19:52:25.409 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57e98a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:25.409 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57e98a8640 1 -- 172.21.15.81:0/1211731354 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e41382a0 con 0x7f57e40a98f0 2024-03-25T19:52:25.409 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57e98a8640 10 client.4687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:25.409 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57e98a8640 10 monclient: _renew_subs 2024-03-25T19:52:25.409 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57e98a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:25.409 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57e98a8640 1 -- 172.21.15.81:0/1211731354 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e41db170 con 0x7f57e40a98f0 2024-03-25T19:52:25.409 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.408+0000 7f57e98a8640 1 librados: init done 2024-03-25T19:52:25.409 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.409+0000 7f57e98a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:25.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.409+0000 7f57e98a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:25.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.409+0000 7f57e98a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:25.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.409+0000 7f57e98a8640 1 -- 172.21.15.81:0/1211731354 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e41db420 con 0x7f57e40a98f0 2024-03-25T19:52:25.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.630+0000 7f57cbfff640 1 -- 172.21.15.81:0/1211731354 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57d4003610 con 0x7f57e40a98f0 2024-03-25T19:52:25.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.630+0000 7f57cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:25.630 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.630+0000 7f57e17fa640 4 set_mon_vals no callback set 2024-03-25T19:52:25.670 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.671+0000 7f57cbfff640 1 -- 172.21.15.81:0/1211731354 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f57d4001d80 con 0x7f57e40a98f0 2024-03-25T19:52:25.670 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.671+0000 7f57cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:25.670 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.671+0000 7f57cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:25.671 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.671+0000 7f57cbfff640 10 monclient: dump: 2024-03-25T19:52:25.671 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:25.671 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:25.671 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:25.671 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:25.671 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:25.671 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:25.671 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:25.671 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:25.671 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:25.671 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:25.679 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.679+0000 7f57cbfff640 1 -- 172.21.15.81:0/1211731354 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f57d4003a20 con 0x7f57e40a98f0 2024-03-25T19:52:25.679 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.680+0000 7f57e27fc640 10 monclient: get_auth_request con 0x7f57d006dcf0 auth_method 0 2024-03-25T19:52:25.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.680+0000 7f57cbfff640 10 client.4687.objecter ms_handle_connect 0x7f57d006dcf0 2024-03-25T19:52:25.681 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.682+0000 7f57cbfff640 1 -- 172.21.15.81:0/1211731354 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3447814714 0 0) 0x7f57d40535a0 con 0x7f57e40a98f0 2024-03-25T19:52:25.681 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.682+0000 7f57cbfff640 10 client.4687.objecter ms_dispatch 0x7f57e4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:25.681 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.682+0000 7f57cbfff640 3 client.4687.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:25.681 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.682+0000 7f57cbfff640 3 client.4687.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:25.682 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.682+0000 7f57cbfff640 20 client.4687.objecter dump_active .. 0 homeless 2024-03-25T19:52:25.686 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.687+0000 7f57cbfff640 1 -- 172.21.15.81:0/1211731354 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f57d404f0b0 con 0x7f57e40a98f0 2024-03-25T19:52:25.686 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.687+0000 7f57cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:25.686 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.687+0000 7f57cbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:25.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.893+0000 7f57e98a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:25.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.893+0000 7f57e98a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:25.893 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.893+0000 7f57e98a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:25.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.893+0000 7f57e98a8640 1 -- 172.21.15.81:0/1211731354 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57e40b3ef0 con 0x7f57e40a98f0 2024-03-25T19:52:25.941 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.942+0000 7f57cbfff640 1 -- 172.21.15.81:0/1211731354 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3235219182) 0x7f57d4050fe0 con 0x7f57e40a98f0 2024-03-25T19:52:25.942 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.942+0000 7f57cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:25.942 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.942+0000 7f57cbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:25.942 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:52:25.942 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":11,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:52:03.690842+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:52:03.690842+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:25.944 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.944+0000 7f57e98a8640 10 librados: watch_flush enter 2024-03-25T19:52:25.944 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.944+0000 7f57e98a8640 10 librados: watch_flush exit 2024-03-25T19:52:25.944 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.944+0000 7f57e98a8640 20 client.4687.objecter shutdown clearing up homeless session... 2024-03-25T19:52:25.945 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.944+0000 7f57e98a8640 10 client.4687.objecter successfully canceled tick 2024-03-25T19:52:25.945 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.944+0000 7f57e98a8640 1 -- 172.21.15.81:0/1211731354 >> v1:172.21.15.98:6800/3980829775 conn(0x7f57d006dcf0 legacy=0x7f57d00701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:25.945 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.944+0000 7f57e98a8640 10 monclient: shutdown 2024-03-25T19:52:25.945 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.944+0000 7f57e98a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:25.945 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.944+0000 7f57e98a8640 1 -- 172.21.15.81:0/1211731354 >> v1:172.21.15.98:6789/0 conn(0x7f57e40a98f0 legacy=0x7f57e4137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:25.945 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.944+0000 7f57e98a8640 1 -- 172.21.15.81:0/1211731354 shutdown_connections 2024-03-25T19:52:25.945 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.944+0000 7f57e98a8640 1 -- 172.21.15.81:0/1211731354 >> 172.21.15.81:0/1211731354 conn(0x7f57e40b7ea0 msgr2=0x7f57e4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:25.945 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.944+0000 7f57e98a8640 1 -- 172.21.15.81:0/1211731354 shutdown_connections 2024-03-25T19:52:25.945 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.944+0000 7f57e98a8640 1 -- 172.21.15.81:0/1211731354 wait complete. 2024-03-25T19:52:25.945 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:25.944+0000 7f57e98a8640 1 librados: shutdown 2024-03-25T19:52:25.956 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:28.814 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:10:51.456 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:28.856 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:51.491 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:28.859 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:51.494 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:28.865 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:10:51.507 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:28.902 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:10:51.536 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:28.904 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:10:51.546 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:28.912 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:10:51.554 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:28.930 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:10:51.572 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:28.966 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:10:51.608 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:32.763 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:32.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.884+0000 7f4be980f640 10 monclient: get_monmap_and_config 2024-03-25T19:52:32.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.884+0000 7f4be980f640 10 monclient: build_initial_monmap 2024-03-25T19:52:32.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.884+0000 7f4be980f640 10 monclient: monmap: 2024-03-25T19:52:32.884 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:52:32.884 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:32.884 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:52:32.885107+0000 2024-03-25T19:52:32.884 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:52:32.885107+0000 2024-03-25T19:52:32.884 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:32.884 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:32.884 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:52:32.884 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:52:32.884 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:52:32.884 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:32.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.885+0000 7f4be980f640 1 Processor -- start 2024-03-25T19:52:32.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.885+0000 7f4be980f640 1 -- start start 2024-03-25T19:52:32.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.885+0000 7f4be980f640 10 monclient: init 2024-03-25T19:52:32.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.885+0000 7f4be980f640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:32.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.885+0000 7f4be980f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:52:32.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.885+0000 7f4be980f640 10 monclient(hunting): picked mon.noname-a con 0x7f4be40a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:52:32.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.885+0000 7f4be980f640 10 monclient(hunting): picked mon.noname-b con 0x7f4be413f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:52:32.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.885+0000 7f4be980f640 10 monclient(hunting): picked mon.noname-c con 0x7f4be41430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:52:32.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.885+0000 7f4be980f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be4142520 con 0x7f4be40a98f0 2024-03-25T19:52:32.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.885+0000 7f4be980f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be40b28a0 con 0x7f4be41430c0 2024-03-25T19:52:32.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.885+0000 7f4be980f640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be40b2a30 con 0x7f4be413f710 2024-03-25T19:52:32.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.885+0000 7f4be980f640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:32.886 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.885+0000 7f4be980f640 10 monclient(hunting): authenticate will time out at 4587.967804s 2024-03-25T19:52:32.894 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.895+0000 7f4be2ffd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f4be40a98f0 0x7f4be40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:49742/0 (socket says 172.21.15.81:49742) 2024-03-25T19:52:32.895 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.895+0000 7f4be2ffd640 1 -- 172.21.15.81:0/1801023920 learned_addr learned my addr 172.21.15.81:0/1801023920 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:52:32.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.911+0000 7f4be27fc640 1 -- 172.21.15.81:0/1801023920 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2088684599 0 0) 0x7f4be40b28a0 con 0x7f4be41430c0 2024-03-25T19:52:32.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.911+0000 7f4be27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:32.910 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.911+0000 7f4be27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:32.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.911+0000 7f4be27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:32.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.911+0000 7f4be27fc640 10 monclient(hunting): my global_id is 4690 2024-03-25T19:52:32.911 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.911+0000 7f4be27fc640 1 -- 172.21.15.81:0/1801023920 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bd0003520 con 0x7f4be41430c0 2024-03-25T19:52:32.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.912+0000 7f4be27fc640 1 -- 172.21.15.81:0/1801023920 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383325063 0 0) 0x7f4be4142520 con 0x7f4be40a98f0 2024-03-25T19:52:32.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.912+0000 7f4be27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:32.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.912+0000 7f4be27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:32.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.912+0000 7f4be27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:32.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.912+0000 7f4be27fc640 10 monclient(hunting): my global_id is 4713 2024-03-25T19:52:32.912 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.912+0000 7f4be27fc640 1 -- 172.21.15.81:0/1801023920 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4be40b28a0 con 0x7f4be40a98f0 2024-03-25T19:52:32.919 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.920+0000 7f4be27fc640 1 -- 172.21.15.81:0/1801023920 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189269858 0 0) 0x7f4be40b2a30 con 0x7f4be413f710 2024-03-25T19:52:32.919 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.920+0000 7f4be27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:32.920 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.920+0000 7f4be27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:32.920 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.920+0000 7f4be27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:32.920 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.920+0000 7f4be27fc640 10 monclient(hunting): my global_id is 4673 2024-03-25T19:52:32.920 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.920+0000 7f4be27fc640 1 -- 172.21.15.81:0/1801023920 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4be4142520 con 0x7f4be413f710 2024-03-25T19:52:32.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.927+0000 7f4be27fc640 1 -- 172.21.15.81:0/1801023920 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3639860905 0 0) 0x7f4be40b28a0 con 0x7f4be40a98f0 2024-03-25T19:52:32.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.927+0000 7f4be27fc640 1 -- 172.21.15.81:0/1801023920 >> v1:172.21.15.81:6790/0 conn(0x7f4be413f710 legacy=0x7f4be41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:32.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.927+0000 7f4be27fc640 1 -- 172.21.15.81:0/1801023920 >> v1:172.21.15.98:6789/0 conn(0x7f4be41430c0 legacy=0x7f4be414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:32.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.927+0000 7f4be27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:52:32.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.927+0000 7f4be27fc640 1 monclient: found mon.noname-a 2024-03-25T19:52:32.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.927+0000 7f4be27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:32.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.927+0000 7f4be27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:52:32.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.927+0000 7f4be27fc640 1 -- 172.21.15.81:0/1801023920 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be40b44c0 con 0x7f4be40a98f0 2024-03-25T19:52:32.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.927+0000 7f4be27fc640 10 monclient: _finish_auth 0 2024-03-25T19:52:32.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.927+0000 7f4be27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:52:32.927 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.927+0000 7f4be27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:32.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.927+0000 7f4be27fc640 1 -- 172.21.15.81:0/1801023920 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4bd4001670 con 0x7f4be40a98f0 2024-03-25T19:52:32.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.927+0000 7f4be27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:32.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.927+0000 7f4be27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:32.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.927+0000 7f4be27fc640 10 monclient: dump: 2024-03-25T19:52:32.928 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:32.928 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:32.928 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:32.928 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:32.928 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:32.928 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:32.928 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:32.928 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:32.928 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:32.928 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:32.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.927+0000 7f4be980f640 5 monclient: authenticate success, global_id 4713 2024-03-25T19:52:32.928 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.927+0000 7f4be980f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:32.946 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.947+0000 7f4be27fc640 1 -- 172.21.15.81:0/1801023920 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bd40017d0 con 0x7f4be40a98f0 2024-03-25T19:52:32.946 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.947+0000 7f4be27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:32.946 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.947+0000 7f4be27fc640 1 -- 172.21.15.81:0/1801023920 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4bd4001be0 con 0x7f4be40a98f0 2024-03-25T19:52:32.946 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.947+0000 7f4be980f640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:32.947 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.947+0000 7f4be980f640 4 set_mon_vals no callback set 2024-03-25T19:52:32.947 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.947+0000 7f4be27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:32.947 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.947+0000 7f4be27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:32.947 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.947+0000 7f4be27fc640 10 monclient: dump: 2024-03-25T19:52:32.947 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:32.947 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:32.947 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:32.947 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:32.947 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:32.947 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:32.947 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:32.947 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:32.947 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:32.947 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:32.949 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.947+0000 7f4be980f640 10 monclient: shutdown 2024-03-25T19:52:32.949 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.947+0000 7f4be980f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:32.949 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.947+0000 7f4be980f640 1 -- 172.21.15.81:0/1801023920 >> v1:172.21.15.81:6789/0 conn(0x7f4be40a98f0 legacy=0x7f4be40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:32.949 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.948+0000 7f4be980f640 1 -- 172.21.15.81:0/1801023920 shutdown_connections 2024-03-25T19:52:32.949 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.948+0000 7f4be980f640 1 -- 172.21.15.81:0/1801023920 >> 172.21.15.81:0/1801023920 conn(0x7f4be40b7ea0 msgr2=0x7f4be40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:32.949 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.948+0000 7f4be980f640 1 -- 172.21.15.81:0/1801023920 shutdown_connections 2024-03-25T19:52:32.949 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.948+0000 7f4be980f640 1 -- 172.21.15.81:0/1801023920 wait complete. 2024-03-25T19:52:32.949 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.948+0000 7f4be980f640 10 monclient: build_initial_monmap 2024-03-25T19:52:32.949 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.948+0000 7f4be980f640 10 monclient: monmap: 2024-03-25T19:52:32.949 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:52:32.949 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:32.949 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:52:32.949 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:52:32.949 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:32.949 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:52:32.950 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:52:32.950 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:52:32.950 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:52:32.950 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:32.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.949+0000 7f4be980f640 1 librados: starting msgr at 2024-03-25T19:52:32.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.949+0000 7f4be980f640 1 librados: starting objecter 2024-03-25T19:52:32.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.949+0000 7f4be980f640 1 Processor -- start 2024-03-25T19:52:32.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.949+0000 7f4be980f640 1 -- start start 2024-03-25T19:52:32.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.949+0000 7f4be980f640 1 librados: setting wanted keys 2024-03-25T19:52:32.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.949+0000 7f4be980f640 1 librados: calling monclient init 2024-03-25T19:52:32.950 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.949+0000 7f4be980f640 10 monclient: init 2024-03-25T19:52:32.951 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.949+0000 7f4be980f640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:32.951 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.949+0000 7f4be980f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:52:32.951 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.949+0000 7f4be980f640 10 monclient(hunting): picked mon.noname-a con 0x7f4be40a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:52:32.951 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.949+0000 7f4be980f640 10 monclient(hunting): picked mon.noname-b con 0x7f4be413f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:52:32.951 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.949+0000 7f4be980f640 10 monclient(hunting): picked mon.noname-c con 0x7f4be41430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:52:32.951 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.949+0000 7f4be980f640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be41dad60 con 0x7f4be40a98f0 2024-03-25T19:52:32.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.950+0000 7f4be980f640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be40b1730 con 0x7f4be413f710 2024-03-25T19:52:32.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.950+0000 7f4be980f640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be414de50 con 0x7f4be41430c0 2024-03-25T19:52:32.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.950+0000 7f4be980f640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:32.952 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.950+0000 7f4be980f640 10 monclient(hunting): authenticate will time out at 4588.031873s 2024-03-25T19:52:32.956 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.957+0000 7f4bdbfff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4be413f710 0x7f4be40b1020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:48086/0 (socket says 172.21.15.81:48086) 2024-03-25T19:52:32.956 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.957+0000 7f4bdbfff640 1 -- 172.21.15.81:0/3809823336 learned_addr learned my addr 172.21.15.81:0/3809823336 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:52:32.957 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.958+0000 7f4be0ff9640 10 client.?.objecter ms_handle_connect 0x7f4be413f710 2024-03-25T19:52:32.957 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.958+0000 7f4be0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:32.962 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.963+0000 7f4be0ff9640 10 client.?.objecter ms_handle_connect 0x7f4be41430c0 2024-03-25T19:52:32.962 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.963+0000 7f4be0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:32.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.966+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843039668 0 0) 0x7f4be40b1730 con 0x7f4be413f710 2024-03-25T19:52:32.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.966+0000 7f4be0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:52:32.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.966+0000 7f4be0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:32.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.966+0000 7f4be0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:32.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.966+0000 7f4be0ff9640 10 monclient(hunting): my global_id is 4693 2024-03-25T19:52:32.966 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.966+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bc4004eb0 con 0x7f4be413f710 2024-03-25T19:52:32.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.968+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056155743 0 0) 0x7f4be414de50 con 0x7f4be41430c0 2024-03-25T19:52:32.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.968+0000 7f4be0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:52:32.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.968+0000 7f4be0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:32.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.968+0000 7f4be0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:32.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.968+0000 7f4be0ff9640 10 monclient(hunting): my global_id is 4676 2024-03-25T19:52:32.967 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.968+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4be40b1730 con 0x7f4be41430c0 2024-03-25T19:52:32.968 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.969+0000 7f4be0ff9640 10 client.?.objecter ms_handle_connect 0x7f4be40a98f0 2024-03-25T19:52:32.969 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.969+0000 7f4be0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:32.980 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.980+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924390474 0 0) 0x7f4be41dad60 con 0x7f4be40a98f0 2024-03-25T19:52:32.980 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.980+0000 7f4be0ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:52:32.980 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.980+0000 7f4be0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:32.980 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.980+0000 7f4be0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:32.980 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.980+0000 7f4be0ff9640 10 monclient(hunting): my global_id is 4716 2024-03-25T19:52:32.980 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.980+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4be414de50 con 0x7f4be40a98f0 2024-03-25T19:52:32.983 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.983+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2367581740 0 0) 0x7f4bc4004eb0 con 0x7f4be413f710 2024-03-25T19:52:32.983 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.983+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4be41dad60 con 0x7f4be413f710 2024-03-25T19:52:32.983 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.983+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4bcc001a10 con 0x7f4be413f710 2024-03-25T19:52:32.983 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.983+0000 7f4be0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:32.983 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.983+0000 7f4be0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:32.983 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.983+0000 7f4be0ff9640 10 monclient(hunting): dump: 2024-03-25T19:52:32.983 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:32.983 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:32.983 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:32.983 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:32.984 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:32.984 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:32.984 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:32.984 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:32.984 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:32.984 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:32.984 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.983+0000 7f4be0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:32.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.990+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3132280264 0 0) 0x7f4be414de50 con 0x7f4be40a98f0 2024-03-25T19:52:32.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.990+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4bc4004eb0 con 0x7f4be40a98f0 2024-03-25T19:52:32.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.990+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4bd4001ae0 con 0x7f4be40a98f0 2024-03-25T19:52:32.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.990+0000 7f4be0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:32.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.990+0000 7f4be0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:32.990 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.990+0000 7f4be0ff9640 10 monclient(hunting): dump: 2024-03-25T19:52:32.990 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:32.990 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:32.990 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:32.990 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:32.990 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:32.991 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:32.991 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:32.991 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:32.991 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:32.991 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:32.994 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.994+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 761668979 0 0) 0x7f4be40b1730 con 0x7f4be41430c0 2024-03-25T19:52:32.994 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.994+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4be414de50 con 0x7f4be41430c0 2024-03-25T19:52:32.994 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.994+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4bdc001af0 con 0x7f4be41430c0 2024-03-25T19:52:32.994 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.994+0000 7f4be0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:32.994 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.994+0000 7f4be0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:52:32.994 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:32.994+0000 7f4be0ff9640 10 monclient(hunting): dump: 2024-03-25T19:52:32.994 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:32.995 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:32.995 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:32.995 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:32.995 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:32.995 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:32.995 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:32.995 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:32.995 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:32.995 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:33.000 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 834123342 0 0) 0x7f4be41dad60 con 0x7f4be413f710 2024-03-25T19:52:33.000 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 >> v1:172.21.15.81:6790/0 conn(0x7f4be41430c0 legacy=0x7f4be41e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:33.000 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 >> v1:172.21.15.81:6789/0 conn(0x7f4be40a98f0 legacy=0x7f4be41db1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:33.000 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be0ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:52:33.000 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be0ff9640 1 monclient: found mon.b 2024-03-25T19:52:33.000 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:33.000 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:33.000 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be414eb60 con 0x7f4be413f710 2024-03-25T19:52:33.000 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be0ff9640 10 monclient: _finish_auth 0 2024-03-25T19:52:33.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be0ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:52:33.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:33.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be980f640 5 monclient: authenticate success, global_id 4693 2024-03-25T19:52:33.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be980f640 10 monclient: _renew_subs 2024-03-25T19:52:33.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be980f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:33.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be980f640 1 -- 172.21.15.81:0/3809823336 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be41da8f0 con 0x7f4be413f710 2024-03-25T19:52:33.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be980f640 10 client.4693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:33.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be980f640 10 monclient: _renew_subs 2024-03-25T19:52:33.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be980f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:33.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be980f640 1 -- 172.21.15.81:0/3809823336 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be414e2a0 con 0x7f4be413f710 2024-03-25T19:52:33.001 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.000+0000 7f4be980f640 1 librados: init done 2024-03-25T19:52:33.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.001+0000 7f4be980f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:33.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.001+0000 7f4be980f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:33.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.001+0000 7f4be980f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:33.002 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.001+0000 7f4be980f640 1 -- 172.21.15.81:0/3809823336 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be40b0a40 con 0x7f4be413f710 2024-03-25T19:52:33.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.282+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bcc001c70 con 0x7f4be413f710 2024-03-25T19:52:33.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.282+0000 7f4be0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:33.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.282+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4bcc003a60 con 0x7f4be413f710 2024-03-25T19:52:33.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.282+0000 7f4be0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:33.283 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.282+0000 7f4be0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:33.283 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.282+0000 7f4be1ffb640 4 set_mon_vals no callback set 2024-03-25T19:52:33.283 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.282+0000 7f4be0ff9640 10 monclient: dump: 2024-03-25T19:52:33.283 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:33.283 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:33.283 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:33.283 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:33.283 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:33.283 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:33.283 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:33.283 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:33.283 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:33.283 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:33.284 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.283+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f4bcc003c80 con 0x7f4be413f710 2024-03-25T19:52:33.284 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.284+0000 7f4be2ffd640 10 monclient: get_auth_request con 0x7f4bc406e0f0 auth_method 0 2024-03-25T19:52:33.284 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.284+0000 7f4be0ff9640 10 client.4693.objecter ms_handle_connect 0x7f4bc406e0f0 2024-03-25T19:52:33.287 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.288+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3447814714 0 0) 0x7f4bcc0548b0 con 0x7f4be413f710 2024-03-25T19:52:33.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.288+0000 7f4be0ff9640 10 client.4693.objecter ms_dispatch 0x7f4be4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:33.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.288+0000 7f4be0ff9640 3 client.4693.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:33.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.288+0000 7f4be0ff9640 3 client.4693.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:33.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.288+0000 7f4be0ff9640 20 client.4693.objecter dump_active .. 0 homeless 2024-03-25T19:52:33.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.288+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4bcc0503c0 con 0x7f4be413f710 2024-03-25T19:52:33.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.288+0000 7f4be0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:33.288 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.288+0000 7f4be0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:33.538 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.539+0000 7f4be980f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:33.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.539+0000 7f4be980f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:33.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.539+0000 7f4be980f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:33.539 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.539+0000 7f4be980f640 1 -- 172.21.15.81:0/3809823336 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4be41daf10 con 0x7f4be413f710 2024-03-25T19:52:33.632 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.632+0000 7f4be0ff9640 1 -- 172.21.15.81:0/3809823336 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3235219182) 0x7f4be41daf10 con 0x7f4be413f710 2024-03-25T19:52:33.632 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.632+0000 7f4be0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:33.632 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.632+0000 7f4be0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:33.632 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:52:33.632 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":11,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:52:03.690842+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:52:03.690842+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:33.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.634+0000 7f4be980f640 10 librados: watch_flush enter 2024-03-25T19:52:33.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.635+0000 7f4be980f640 10 librados: watch_flush exit 2024-03-25T19:52:33.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.635+0000 7f4be980f640 20 client.4693.objecter shutdown clearing up homeless session... 2024-03-25T19:52:33.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.635+0000 7f4be980f640 10 client.4693.objecter successfully canceled tick 2024-03-25T19:52:33.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.635+0000 7f4be980f640 1 -- 172.21.15.81:0/3809823336 >> v1:172.21.15.98:6800/3980829775 conn(0x7f4bc406e0f0 legacy=0x7f4bc40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:33.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.635+0000 7f4be980f640 10 monclient: shutdown 2024-03-25T19:52:33.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.635+0000 7f4be980f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:33.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.635+0000 7f4be980f640 1 -- 172.21.15.81:0/3809823336 >> v1:172.21.15.98:6789/0 conn(0x7f4be413f710 legacy=0x7f4be40b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:33.635 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.635+0000 7f4be980f640 1 -- 172.21.15.81:0/3809823336 shutdown_connections 2024-03-25T19:52:33.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.635+0000 7f4be980f640 1 -- 172.21.15.81:0/3809823336 >> 172.21.15.81:0/3809823336 conn(0x7f4be40b7ea0 msgr2=0x7f4be4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:33.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.635+0000 7f4be980f640 1 -- 172.21.15.81:0/3809823336 shutdown_connections 2024-03-25T19:52:33.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.635+0000 7f4be980f640 1 -- 172.21.15.81:0/3809823336 wait complete. 2024-03-25T19:52:33.636 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:33.635+0000 7f4be980f640 1 librados: shutdown 2024-03-25T19:52:33.646 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:40.453 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:40.562 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.563+0000 7fc5a96ef640 10 monclient: get_monmap_and_config 2024-03-25T19:52:40.563 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.563+0000 7fc5a96ef640 10 monclient: build_initial_monmap 2024-03-25T19:52:40.563 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.563+0000 7fc5a96ef640 10 monclient: monmap: 2024-03-25T19:52:40.563 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:52:40.563 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:40.563 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:52:40.564233+0000 2024-03-25T19:52:40.563 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:52:40.564233+0000 2024-03-25T19:52:40.563 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:40.563 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:40.563 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:52:40.563 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:52:40.563 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:52:40.563 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:40.564 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.563+0000 7fc5a96ef640 1 Processor -- start 2024-03-25T19:52:40.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.563+0000 7fc5a96ef640 1 -- start start 2024-03-25T19:52:40.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.563+0000 7fc5a96ef640 10 monclient: init 2024-03-25T19:52:40.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.564+0000 7fc5a96ef640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:40.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.564+0000 7fc5a96ef640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:52:40.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.564+0000 7fc5a96ef640 10 monclient(hunting): picked mon.noname-c con 0x7fc5a40b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:52:40.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.564+0000 7fc5a96ef640 10 monclient(hunting): picked mon.noname-a con 0x7fc5a40b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:52:40.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.564+0000 7fc5a96ef640 10 monclient(hunting): picked mon.noname-b con 0x7fc5a414be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:52:40.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.564+0000 7fc5a96ef640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a40af940 con 0x7fc5a40b1730 2024-03-25T19:52:40.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.564+0000 7fc5a96ef640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a40b0a70 con 0x7fc5a40b4d60 2024-03-25T19:52:40.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.564+0000 7fc5a96ef640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a40b0d50 con 0x7fc5a414be90 2024-03-25T19:52:40.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.564+0000 7fc5a96ef640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:40.565 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.564+0000 7fc5a96ef640 10 monclient(hunting): authenticate will time out at 4595.646391s 2024-03-25T19:52:40.584 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.584+0000 7fc5a2ffd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fc5a40b4d60 0x7fc5a413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:50654/0 (socket says 172.21.15.81:50654) 2024-03-25T19:52:40.584 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.584+0000 7fc5a2ffd640 1 -- 172.21.15.81:0/2424557841 learned_addr learned my addr 172.21.15.81:0/2424557841 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:52:40.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.606+0000 7fc5a1ffb640 1 -- 172.21.15.81:0/2424557841 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236086661 0 0) 0x7fc5a40b0d50 con 0x7fc5a414be90 2024-03-25T19:52:40.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.606+0000 7fc5a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:52:40.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.606+0000 7fc5a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:40.605 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.606+0000 7fc5a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:40.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.606+0000 7fc5a1ffb640 10 monclient(hunting): my global_id is 4679 2024-03-25T19:52:40.606 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.606+0000 7fc5a1ffb640 1 -- 172.21.15.81:0/2424557841 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc58c003520 con 0x7fc5a414be90 2024-03-25T19:52:40.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.609+0000 7fc5a1ffb640 1 -- 172.21.15.81:0/2424557841 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373332031 0 0) 0x7fc5a40b0a70 con 0x7fc5a40b4d60 2024-03-25T19:52:40.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.609+0000 7fc5a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:52:40.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.609+0000 7fc5a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:40.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.609+0000 7fc5a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:40.608 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.609+0000 7fc5a1ffb640 10 monclient(hunting): my global_id is 4696 2024-03-25T19:52:40.609 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.609+0000 7fc5a1ffb640 1 -- 172.21.15.81:0/2424557841 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a40b0d50 con 0x7fc5a40b4d60 2024-03-25T19:52:40.632 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a1ffb640 1 -- 172.21.15.81:0/2424557841 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2765991563 0 0) 0x7fc5a40b0d50 con 0x7fc5a40b4d60 2024-03-25T19:52:40.632 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a1ffb640 1 -- 172.21.15.81:0/2424557841 >> v1:172.21.15.81:6790/0 conn(0x7fc5a414be90 legacy=0x7fc5a414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:40.632 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a1ffb640 1 -- 172.21.15.81:0/2424557841 >> v1:172.21.15.81:6789/0 conn(0x7fc5a40b1730 legacy=0x7fc5a40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:40.632 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:52:40.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a1ffb640 1 monclient: found mon.noname-c 2024-03-25T19:52:40.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:40.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:52:40.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a1ffb640 1 -- 172.21.15.81:0/2424557841 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a40b1030 con 0x7fc5a40b4d60 2024-03-25T19:52:40.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:52:40.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:52:40.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:40.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a1ffb640 1 -- 172.21.15.81:0/2424557841 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fc594001670 con 0x7fc5a40b4d60 2024-03-25T19:52:40.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a96ef640 5 monclient: authenticate success, global_id 4696 2024-03-25T19:52:40.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a96ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:40.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:40.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:52:40.633 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a1ffb640 10 monclient: dump: 2024-03-25T19:52:40.633 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:40.633 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:40.633 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:40.634 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:40.634 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:40.634 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:40.634 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:40.634 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:40.634 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:40.634 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:40.634 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.632+0000 7fc5a96ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:40.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.641+0000 7fc5a1ffb640 1 -- 172.21.15.81:0/2424557841 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5940017d0 con 0x7fc5a40b4d60 2024-03-25T19:52:40.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.641+0000 7fc5a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:40.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.641+0000 7fc5a1ffb640 1 -- 172.21.15.81:0/2424557841 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fc594001ba0 con 0x7fc5a40b4d60 2024-03-25T19:52:40.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.641+0000 7fc5a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:40.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.641+0000 7fc5a1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:40.641 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.641+0000 7fc5a1ffb640 10 monclient: dump: 2024-03-25T19:52:40.641 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:40.641 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:40.641 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:40.641 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:40.641 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:40.641 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:40.641 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:40.642 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:40.642 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:40.642 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:40.642 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.641+0000 7fc5a96ef640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:40.642 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.641+0000 7fc5a96ef640 4 set_mon_vals no callback set 2024-03-25T19:52:40.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.642+0000 7fc5a96ef640 10 monclient: shutdown 2024-03-25T19:52:40.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.642+0000 7fc5a96ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:40.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.642+0000 7fc5a96ef640 1 -- 172.21.15.81:0/2424557841 >> v1:172.21.15.98:6789/0 conn(0x7fc5a40b4d60 legacy=0x7fc5a413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:40.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.642+0000 7fc5a96ef640 1 -- 172.21.15.81:0/2424557841 shutdown_connections 2024-03-25T19:52:40.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.642+0000 7fc5a96ef640 1 -- 172.21.15.81:0/2424557841 >> 172.21.15.81:0/2424557841 conn(0x7fc5a40617a0 msgr2=0x7fc5a4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:40.643 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.642+0000 7fc5a96ef640 1 -- 172.21.15.81:0/2424557841 shutdown_connections 2024-03-25T19:52:40.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.642+0000 7fc5a96ef640 1 -- 172.21.15.81:0/2424557841 wait complete. 2024-03-25T19:52:40.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.643+0000 7fc5a96ef640 10 monclient: build_initial_monmap 2024-03-25T19:52:40.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.643+0000 7fc5a96ef640 10 monclient: monmap: 2024-03-25T19:52:40.644 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:52:40.644 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:40.644 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:52:40.644 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:52:40.644 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:40.644 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:52:40.644 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:52:40.644 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:52:40.644 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:52:40.644 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:40.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.643+0000 7fc5a96ef640 1 librados: starting msgr at 2024-03-25T19:52:40.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.643+0000 7fc5a96ef640 1 librados: starting objecter 2024-03-25T19:52:40.644 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.643+0000 7fc5a96ef640 1 Processor -- start 2024-03-25T19:52:40.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.643+0000 7fc5a96ef640 1 -- start start 2024-03-25T19:52:40.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.643+0000 7fc5a96ef640 1 librados: setting wanted keys 2024-03-25T19:52:40.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.643+0000 7fc5a96ef640 1 librados: calling monclient init 2024-03-25T19:52:40.645 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.643+0000 7fc5a96ef640 10 monclient: init 2024-03-25T19:52:40.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.644+0000 7fc5a96ef640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:40.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.644+0000 7fc5a96ef640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:52:40.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.644+0000 7fc5a96ef640 10 monclient(hunting): picked mon.noname-b con 0x7fc5a40b1730 addr v1:172.21.15.98:6789/0 2024-03-25T19:52:40.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.644+0000 7fc5a96ef640 10 monclient(hunting): picked mon.noname-c con 0x7fc5a40b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:52:40.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.644+0000 7fc5a96ef640 10 monclient(hunting): picked mon.noname-a con 0x7fc5a414be90 addr v1:172.21.15.81:6789/0 2024-03-25T19:52:40.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.644+0000 7fc5a96ef640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a41406b0 con 0x7fc5a414be90 2024-03-25T19:52:40.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.644+0000 7fc5a96ef640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a41435c0 con 0x7fc5a40b1730 2024-03-25T19:52:40.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.644+0000 7fc5a96ef640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5a4140380 con 0x7fc5a40b4d60 2024-03-25T19:52:40.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.644+0000 7fc5a96ef640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:40.646 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.644+0000 7fc5a96ef640 10 monclient(hunting): authenticate will time out at 4595.726599s 2024-03-25T19:52:40.651 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.652+0000 7fc5a37fe640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fc5a414be90 0x7fc5a41e2320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:35120/0 (socket says 172.21.15.81:35120) 2024-03-25T19:52:40.651 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.652+0000 7fc5a37fe640 1 -- 172.21.15.81:0/157294430 learned_addr learned my addr 172.21.15.81:0/157294430 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:52:40.653 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.654+0000 7fc58b7fe640 10 client.?.objecter ms_handle_connect 0x7fc5a414be90 2024-03-25T19:52:40.653 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.654+0000 7fc58b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:40.658 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.658+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260166046 0 0) 0x7fc5a41406b0 con 0x7fc5a414be90 2024-03-25T19:52:40.658 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.658+0000 7fc58b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:52:40.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.658+0000 7fc58b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:40.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.658+0000 7fc58b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:40.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.658+0000 7fc58b7fe640 10 monclient(hunting): my global_id is 4722 2024-03-25T19:52:40.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.659+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc590004eb0 con 0x7fc5a414be90 2024-03-25T19:52:40.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.659+0000 7fc58b7fe640 10 client.?.objecter ms_handle_connect 0x7fc5a40b4d60 2024-03-25T19:52:40.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.659+0000 7fc58b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:40.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.659+0000 7fc58b7fe640 10 client.?.objecter ms_handle_connect 0x7fc5a40b1730 2024-03-25T19:52:40.659 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.659+0000 7fc58b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:40.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.662+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501348959 0 0) 0x7fc5a41435c0 con 0x7fc5a40b1730 2024-03-25T19:52:40.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.662+0000 7fc58b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:52:40.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.662+0000 7fc58b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:40.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.662+0000 7fc58b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:40.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.662+0000 7fc58b7fe640 10 monclient(hunting): my global_id is 4699 2024-03-25T19:52:40.662 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.662+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a41406b0 con 0x7fc5a40b1730 2024-03-25T19:52:40.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.669+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642724988 0 0) 0x7fc5a4140380 con 0x7fc5a40b4d60 2024-03-25T19:52:40.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.669+0000 7fc58b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:52:40.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.669+0000 7fc58b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:40.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.669+0000 7fc58b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:40.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.669+0000 7fc58b7fe640 10 monclient(hunting): my global_id is 4682 2024-03-25T19:52:40.669 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.669+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a41435c0 con 0x7fc5a40b4d60 2024-03-25T19:52:40.671 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.671+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3240652122 0 0) 0x7fc590004eb0 con 0x7fc5a414be90 2024-03-25T19:52:40.671 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.671+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5a4140380 con 0x7fc5a414be90 2024-03-25T19:52:40.671 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.671+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fc59c001c30 con 0x7fc5a414be90 2024-03-25T19:52:40.671 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.671+0000 7fc58b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:40.671 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.671+0000 7fc58b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:40.671 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.671+0000 7fc58b7fe640 10 monclient(hunting): dump: 2024-03-25T19:52:40.671 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:40.671 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:40.671 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:40.671 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:40.671 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:40.671 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:40.671 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:40.671 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:40.671 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:40.671 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:40.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.671+0000 7fc58b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:40.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.673+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3062536858 0 0) 0x7fc5a41406b0 con 0x7fc5a40b1730 2024-03-25T19:52:40.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.673+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc590004eb0 con 0x7fc5a40b1730 2024-03-25T19:52:40.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.673+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fc594001530 con 0x7fc5a40b1730 2024-03-25T19:52:40.672 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.673+0000 7fc58b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:40.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.673+0000 7fc58b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:40.673 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.673+0000 7fc58b7fe640 10 monclient(hunting): dump: 2024-03-25T19:52:40.673 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:40.673 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:40.673 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:40.673 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:40.673 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:40.673 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:40.673 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:40.673 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:40.673 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:40.673 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:40.679 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2399216565 0 0) 0x7fc5a4140380 con 0x7fc5a414be90 2024-03-25T19:52:40.679 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 >> v1:172.21.15.81:6790/0 conn(0x7fc5a40b4d60 legacy=0x7fc5a4141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:40.679 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 >> v1:172.21.15.98:6789/0 conn(0x7fc5a40b1730 legacy=0x7fc5a4140b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:40.679 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc58b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:52:40.679 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc58b7fe640 1 monclient: found mon.a 2024-03-25T19:52:40.679 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc58b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:40.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc58b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:52:40.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a40aaa20 con 0x7fc5a414be90 2024-03-25T19:52:40.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc58b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:52:40.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc58b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:52:40.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc58b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:40.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc5a96ef640 5 monclient: authenticate success, global_id 4722 2024-03-25T19:52:40.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc5a96ef640 10 monclient: _renew_subs 2024-03-25T19:52:40.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc5a96ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:52:40.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc5a96ef640 1 -- 172.21.15.81:0/157294430 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a41430e0 con 0x7fc5a414be90 2024-03-25T19:52:40.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc5a96ef640 10 client.4722.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:40.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc5a96ef640 10 monclient: _renew_subs 2024-03-25T19:52:40.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc5a96ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:52:40.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc5a96ef640 1 -- 172.21.15.81:0/157294430 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a40a98c0 con 0x7fc5a414be90 2024-03-25T19:52:40.680 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.679+0000 7fc5a96ef640 1 librados: init done 2024-03-25T19:52:40.681 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.680+0000 7fc5a96ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:40.681 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.680+0000 7fc5a96ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:40.681 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.680+0000 7fc5a96ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:52:40.681 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.680+0000 7fc5a96ef640 1 -- 172.21.15.81:0/157294430 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a40627b0 con 0x7fc5a414be90 2024-03-25T19:52:40.972 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.972+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc59c0020e0 con 0x7fc5a414be90 2024-03-25T19:52:40.972 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.972+0000 7fc58b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:40.972 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.972+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fc59c003b70 con 0x7fc5a414be90 2024-03-25T19:52:40.972 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.972+0000 7fc58b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:40.972 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.972+0000 7fc58b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:40.972 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.972+0000 7fc58b7fe640 10 monclient: dump: 2024-03-25T19:52:40.972 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:40.972 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:40.972 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:40.972 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:40.972 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:40.973 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:40.973 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:40.973 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:40.973 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:40.973 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:40.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.972+0000 7fc5a17fa640 4 set_mon_vals no callback set 2024-03-25T19:52:40.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.973+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fc59c003d90 con 0x7fc5a414be90 2024-03-25T19:52:40.973 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.974+0000 7fc5a2ffd640 10 monclient: get_auth_request con 0x7fc59006df20 auth_method 0 2024-03-25T19:52:40.974 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.974+0000 7fc58b7fe640 10 client.4722.objecter ms_handle_connect 0x7fc59006df20 2024-03-25T19:52:40.981 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.981+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3447814714 0 0) 0x7fc59c0549b0 con 0x7fc5a414be90 2024-03-25T19:52:40.981 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.981+0000 7fc58b7fe640 10 client.4722.objecter ms_dispatch 0x7fc5a4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:40.981 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.981+0000 7fc58b7fe640 3 client.4722.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:40.981 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.981+0000 7fc58b7fe640 3 client.4722.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:40.981 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.981+0000 7fc58b7fe640 20 client.4722.objecter dump_active .. 0 homeless 2024-03-25T19:52:40.981 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.981+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc59c0502f0 con 0x7fc5a414be90 2024-03-25T19:52:40.981 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.981+0000 7fc58b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:40.981 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:40.981+0000 7fc58b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:41.219 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.220+0000 7fc5a96ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:41.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.220+0000 7fc5a96ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:41.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.220+0000 7fc5a96ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:52:41.220 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.220+0000 7fc5a96ef640 1 -- 172.21.15.81:0/157294430 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5a40b6420 con 0x7fc5a414be90 2024-03-25T19:52:41.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.246+0000 7fc58b7fe640 1 -- 172.21.15.81:0/157294430 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3235219182) 0x7fc59c0523f0 con 0x7fc5a414be90 2024-03-25T19:52:41.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.246+0000 7fc58b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:41.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.246+0000 7fc58b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:41.246 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:52:41.246 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":11,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:52:03.690842+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:52:03.690842+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:41.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.248+0000 7fc5a96ef640 10 librados: watch_flush enter 2024-03-25T19:52:41.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.248+0000 7fc5a96ef640 10 librados: watch_flush exit 2024-03-25T19:52:41.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.248+0000 7fc5a96ef640 20 client.4722.objecter shutdown clearing up homeless session... 2024-03-25T19:52:41.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.248+0000 7fc5a96ef640 10 client.4722.objecter successfully canceled tick 2024-03-25T19:52:41.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.248+0000 7fc5a96ef640 1 -- 172.21.15.81:0/157294430 >> v1:172.21.15.98:6800/3980829775 conn(0x7fc59006df20 legacy=0x7fc5900703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:41.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.248+0000 7fc5a96ef640 10 monclient: shutdown 2024-03-25T19:52:41.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.248+0000 7fc5a96ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:41.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.248+0000 7fc5a96ef640 1 -- 172.21.15.81:0/157294430 >> v1:172.21.15.81:6789/0 conn(0x7fc5a414be90 legacy=0x7fc5a41e2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:41.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.248+0000 7fc5a96ef640 1 -- 172.21.15.81:0/157294430 shutdown_connections 2024-03-25T19:52:41.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.248+0000 7fc5a96ef640 1 -- 172.21.15.81:0/157294430 >> 172.21.15.81:0/157294430 conn(0x7fc5a40617a0 msgr2=0x7fc5a413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:41.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.249+0000 7fc5a96ef640 1 -- 172.21.15.81:0/157294430 shutdown_connections 2024-03-25T19:52:41.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.249+0000 7fc5a96ef640 1 -- 172.21.15.81:0/157294430 wait complete. 2024-03-25T19:52:41.249 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:41.249+0000 7fc5a96ef640 1 librados: shutdown 2024-03-25T19:52:41.260 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:48.066 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:48.185 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.185+0000 7f4d3b767640 10 monclient: get_monmap_and_config 2024-03-25T19:52:48.185 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.185+0000 7f4d3b767640 10 monclient: build_initial_monmap 2024-03-25T19:52:48.185 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.185+0000 7f4d3b767640 10 monclient: monmap: 2024-03-25T19:52:48.185 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:52:48.185 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:48.185 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:52:48.186415+0000 2024-03-25T19:52:48.185 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:52:48.186415+0000 2024-03-25T19:52:48.185 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:48.185 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:48.185 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:52:48.185 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:52:48.185 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:52:48.185 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:48.187 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.186+0000 7f4d3b767640 1 Processor -- start 2024-03-25T19:52:48.187 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.186+0000 7f4d3b767640 1 -- start start 2024-03-25T19:52:48.187 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.186+0000 7f4d3b767640 10 monclient: init 2024-03-25T19:52:48.187 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.187+0000 7f4d3b767640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:48.187 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.187+0000 7f4d3b767640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:52:48.187 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.187+0000 7f4d3b767640 10 monclient(hunting): picked mon.noname-b con 0x7f4d340b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:52:48.187 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.187+0000 7f4d3b767640 10 monclient(hunting): picked mon.noname-a con 0x7f4d340b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:52:48.187 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.187+0000 7f4d3b767640 10 monclient(hunting): picked mon.noname-c con 0x7f4d3414be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:52:48.187 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.187+0000 7f4d3b767640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d340af940 con 0x7f4d340b1730 2024-03-25T19:52:48.187 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.187+0000 7f4d3b767640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d340b0a70 con 0x7f4d3414be90 2024-03-25T19:52:48.187 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.187+0000 7f4d3b767640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d340b0d50 con 0x7f4d340b4d60 2024-03-25T19:52:48.187 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.187+0000 7f4d3b767640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:48.187 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.187+0000 7f4d3b767640 10 monclient(hunting): authenticate will time out at 4603.269126s 2024-03-25T19:52:48.190 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.191+0000 7f4d39d04640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f4d3414be90 0x7f4d3414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:52328/0 (socket says 172.21.15.81:52328) 2024-03-25T19:52:48.190 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.191+0000 7f4d39d04640 1 -- 172.21.15.81:0/2813138888 learned_addr learned my addr 172.21.15.81:0/2813138888 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:52:48.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.201+0000 7f4d2bfff640 1 -- 172.21.15.81:0/2813138888 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743093398 0 0) 0x7f4d340af940 con 0x7f4d340b1730 2024-03-25T19:52:48.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.201+0000 7f4d2bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:48.200 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.201+0000 7f4d2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:48.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.201+0000 7f4d2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:48.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.201+0000 7f4d2bfff640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:52:48.201 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.201+0000 7f4d2bfff640 1 -- 172.21.15.81:0/2813138888 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d20003520 con 0x7f4d340b1730 2024-03-25T19:52:48.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.202+0000 7f4d2bfff640 1 -- 172.21.15.81:0/2813138888 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298764629 0 0) 0x7f4d340b0a70 con 0x7f4d3414be90 2024-03-25T19:52:48.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.202+0000 7f4d2bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:48.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.202+0000 7f4d2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:48.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.202+0000 7f4d2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:48.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.202+0000 7f4d2bfff640 10 monclient(hunting): my global_id is 4702 2024-03-25T19:52:48.202 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.202+0000 7f4d2bfff640 1 -- 172.21.15.81:0/2813138888 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d340af940 con 0x7f4d3414be90 2024-03-25T19:52:48.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d2bfff640 1 -- 172.21.15.81:0/2813138888 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 504073379 0 0) 0x7f4d20003520 con 0x7f4d340b1730 2024-03-25T19:52:48.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d2bfff640 1 -- 172.21.15.81:0/2813138888 >> v1:172.21.15.81:6790/0 conn(0x7f4d340b4d60 legacy=0x7f4d3413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:48.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d2bfff640 1 -- 172.21.15.81:0/2813138888 >> v1:172.21.15.98:6789/0 conn(0x7f4d3414be90 legacy=0x7f4d3414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:48.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d2bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:52:48.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d2bfff640 1 monclient: found mon.noname-a 2024-03-25T19:52:48.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:48.215 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d2bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:52:48.216 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d2bfff640 1 -- 172.21.15.81:0/2813138888 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d340b1030 con 0x7f4d340b1730 2024-03-25T19:52:48.216 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d2bfff640 10 monclient: _finish_auth 0 2024-03-25T19:52:48.216 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d2bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:52:48.216 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:48.216 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d2bfff640 1 -- 172.21.15.81:0/2813138888 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4d24001690 con 0x7f4d340b1730 2024-03-25T19:52:48.216 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d3b767640 5 monclient: authenticate success, global_id 4725 2024-03-25T19:52:48.216 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d3b767640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:48.216 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:48.216 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d2bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:48.216 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d2bfff640 10 monclient: dump: 2024-03-25T19:52:48.216 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:48.216 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:48.216 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:48.216 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:48.216 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:48.216 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:48.217 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:48.217 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:48.217 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:48.217 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:48.217 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.215+0000 7f4d3b767640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:48.222 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.223+0000 7f4d2bfff640 1 -- 172.21.15.81:0/2813138888 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d240017f0 con 0x7f4d340b1730 2024-03-25T19:52:48.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.223+0000 7f4d2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:48.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.223+0000 7f4d2bfff640 1 -- 172.21.15.81:0/2813138888 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4d24001c00 con 0x7f4d340b1730 2024-03-25T19:52:48.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.223+0000 7f4d2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:48.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.223+0000 7f4d2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:48.223 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.223+0000 7f4d2bfff640 10 monclient: dump: 2024-03-25T19:52:48.223 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:48.223 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:48.223 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:48.223 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:48.223 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:48.223 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:48.223 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:48.223 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:48.224 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:48.224 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:48.224 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.223+0000 7f4d3b767640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:48.224 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.223+0000 7f4d3b767640 4 set_mon_vals no callback set 2024-03-25T19:52:48.225 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.224+0000 7f4d3b767640 10 monclient: shutdown 2024-03-25T19:52:48.225 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.224+0000 7f4d3b767640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:48.225 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.224+0000 7f4d3b767640 1 -- 172.21.15.81:0/2813138888 >> v1:172.21.15.81:6789/0 conn(0x7f4d340b1730 legacy=0x7f4d340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:48.225 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.224+0000 7f4d3b767640 1 -- 172.21.15.81:0/2813138888 shutdown_connections 2024-03-25T19:52:48.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.224+0000 7f4d3b767640 1 -- 172.21.15.81:0/2813138888 >> 172.21.15.81:0/2813138888 conn(0x7f4d340617a0 msgr2=0x7f4d34061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:48.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.224+0000 7f4d3b767640 1 -- 172.21.15.81:0/2813138888 shutdown_connections 2024-03-25T19:52:48.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.224+0000 7f4d3b767640 1 -- 172.21.15.81:0/2813138888 wait complete. 2024-03-25T19:52:48.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.225+0000 7f4d3b767640 10 monclient: build_initial_monmap 2024-03-25T19:52:48.226 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.225+0000 7f4d3b767640 10 monclient: monmap: 2024-03-25T19:52:48.226 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:52:48.226 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:48.226 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:52:48.226 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:52:48.226 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:48.226 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:52:48.226 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:52:48.226 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:52:48.226 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:52:48.227 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:48.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.225+0000 7f4d3b767640 1 librados: starting msgr at 2024-03-25T19:52:48.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.225+0000 7f4d3b767640 1 librados: starting objecter 2024-03-25T19:52:48.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.225+0000 7f4d3b767640 1 Processor -- start 2024-03-25T19:52:48.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.225+0000 7f4d3b767640 1 -- start start 2024-03-25T19:52:48.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.226+0000 7f4d3b767640 1 librados: setting wanted keys 2024-03-25T19:52:48.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.226+0000 7f4d3b767640 1 librados: calling monclient init 2024-03-25T19:52:48.227 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.226+0000 7f4d3b767640 10 monclient: init 2024-03-25T19:52:48.228 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.226+0000 7f4d3b767640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:48.228 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.226+0000 7f4d3b767640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:52:48.228 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.226+0000 7f4d3b767640 10 monclient(hunting): picked mon.noname-a con 0x7f4d340b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:52:48.228 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.226+0000 7f4d3b767640 10 monclient(hunting): picked mon.noname-c con 0x7f4d340b4d60 addr v1:172.21.15.81:6790/0 2024-03-25T19:52:48.228 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.226+0000 7f4d3b767640 10 monclient(hunting): picked mon.noname-b con 0x7f4d3414be90 addr v1:172.21.15.98:6789/0 2024-03-25T19:52:48.228 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.226+0000 7f4d3b767640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d34141f10 con 0x7f4d340b1730 2024-03-25T19:52:48.228 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.226+0000 7f4d3b767640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d34140da0 con 0x7f4d3414be90 2024-03-25T19:52:48.228 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.226+0000 7f4d3b767640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d34138860 con 0x7f4d340b4d60 2024-03-25T19:52:48.228 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.226+0000 7f4d3b767640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:48.228 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.226+0000 7f4d3b767640 10 monclient(hunting): authenticate will time out at 4603.308739s 2024-03-25T19:52:48.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.231+0000 7f4d38d02640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f4d340b4d60 0x7f4d341d7970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:34236/0 (socket says 172.21.15.81:34236) 2024-03-25T19:52:48.231 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.231+0000 7f4d38d02640 1 -- 172.21.15.81:0/2771465667 learned_addr learned my addr 172.21.15.81:0/2771465667 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:52:48.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.232+0000 7f4d29ffb640 10 client.?.objecter ms_handle_connect 0x7f4d340b4d60 2024-03-25T19:52:48.232 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.232+0000 7f4d29ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:48.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.238+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408868545 0 0) 0x7f4d34138860 con 0x7f4d340b4d60 2024-03-25T19:52:48.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.239+0000 7f4d29ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:52:48.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.239+0000 7f4d29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:48.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.239+0000 7f4d29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:48.238 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.239+0000 7f4d29ffb640 10 monclient(hunting): my global_id is 4685 2024-03-25T19:52:48.239 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.239+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d18004eb0 con 0x7f4d340b4d60 2024-03-25T19:52:48.242 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.243+0000 7f4d29ffb640 10 client.?.objecter ms_handle_connect 0x7f4d340b1730 2024-03-25T19:52:48.242 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.243+0000 7f4d29ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:48.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.247+0000 7f4d29ffb640 10 client.?.objecter ms_handle_connect 0x7f4d3414be90 2024-03-25T19:52:48.246 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.247+0000 7f4d29ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:48.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.254+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300490162 0 0) 0x7f4d34141f10 con 0x7f4d340b1730 2024-03-25T19:52:48.253 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.254+0000 7f4d29ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:52:48.254 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.254+0000 7f4d29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:48.254 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.254+0000 7f4d29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:48.254 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.254+0000 7f4d29ffb640 10 monclient(hunting): my global_id is 4728 2024-03-25T19:52:48.254 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.254+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d34138860 con 0x7f4d340b1730 2024-03-25T19:52:48.256 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.256+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2722067836 0 0) 0x7f4d18004eb0 con 0x7f4d340b4d60 2024-03-25T19:52:48.256 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.256+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d34141f10 con 0x7f4d340b4d60 2024-03-25T19:52:48.256 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.256+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4d24001bc0 con 0x7f4d340b4d60 2024-03-25T19:52:48.256 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.256+0000 7f4d29ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:48.256 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.256+0000 7f4d29ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:52:48.256 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.256+0000 7f4d29ffb640 10 monclient(hunting): dump: 2024-03-25T19:52:48.256 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:48.256 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:48.256 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:48.256 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:48.256 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:48.257 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:48.257 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:48.257 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:48.257 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:48.257 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:48.257 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.256+0000 7f4d29ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:48.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.259+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641218038 0 0) 0x7f4d34140da0 con 0x7f4d3414be90 2024-03-25T19:52:48.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.259+0000 7f4d29ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:52:48.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.259+0000 7f4d29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:48.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.259+0000 7f4d29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:48.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.259+0000 7f4d29ffb640 10 monclient(hunting): my global_id is 4705 2024-03-25T19:52:48.259 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.259+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d18004eb0 con 0x7f4d3414be90 2024-03-25T19:52:48.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.270+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2072680424 0 0) 0x7f4d34138860 con 0x7f4d340b1730 2024-03-25T19:52:48.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.270+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d34140da0 con 0x7f4d340b1730 2024-03-25T19:52:48.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.270+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4d300018b0 con 0x7f4d340b1730 2024-03-25T19:52:48.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.270+0000 7f4d29ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:48.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.270+0000 7f4d29ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:48.270 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.270+0000 7f4d29ffb640 10 monclient(hunting): dump: 2024-03-25T19:52:48.270 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:48.270 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:48.270 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:48.270 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:48.271 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:48.271 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:48.271 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:48.271 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:48.271 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:48.271 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:48.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.279+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3619218462 0 0) 0x7f4d34140da0 con 0x7f4d340b1730 2024-03-25T19:52:48.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.279+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 >> v1:172.21.15.81:6790/0 conn(0x7f4d340b4d60 legacy=0x7f4d341d7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:48.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.279+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 >> v1:172.21.15.98:6789/0 conn(0x7f4d3414be90 legacy=0x7f4d341e2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:48.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.279+0000 7f4d29ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:52:48.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.279+0000 7f4d29ffb640 1 monclient: found mon.a 2024-03-25T19:52:48.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.279+0000 7f4d29ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:48.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.280+0000 7f4d29ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:52:48.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.280+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d34137f50 con 0x7f4d340b1730 2024-03-25T19:52:48.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.280+0000 7f4d29ffb640 10 monclient: _finish_auth 0 2024-03-25T19:52:48.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.280+0000 7f4d29ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:52:48.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.280+0000 7f4d29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:48.280 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.280+0000 7f4d3b767640 5 monclient: authenticate success, global_id 4728 2024-03-25T19:52:48.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.280+0000 7f4d3b767640 10 monclient: _renew_subs 2024-03-25T19:52:48.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.280+0000 7f4d3b767640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:52:48.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.280+0000 7f4d3b767640 1 -- 172.21.15.81:0/2771465667 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d341d8080 con 0x7f4d340b1730 2024-03-25T19:52:48.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.280+0000 7f4d3b767640 10 client.4728.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:48.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.280+0000 7f4d3b767640 10 monclient: _renew_subs 2024-03-25T19:52:48.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.280+0000 7f4d3b767640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:52:48.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.280+0000 7f4d3b767640 1 -- 172.21.15.81:0/2771465667 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d341d85b0 con 0x7f4d340b1730 2024-03-25T19:52:48.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.280+0000 7f4d3b767640 1 librados: init done 2024-03-25T19:52:48.281 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.281+0000 7f4d3b767640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:48.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.281+0000 7f4d3b767640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:48.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.281+0000 7f4d3b767640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:52:48.282 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.281+0000 7f4d3b767640 1 -- 172.21.15.81:0/2771465667 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d341381a0 con 0x7f4d340b1730 2024-03-25T19:52:48.586 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.586+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d30001a10 con 0x7f4d340b1730 2024-03-25T19:52:48.586 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.586+0000 7f4d29ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:48.586 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.586+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f4d300037d0 con 0x7f4d340b1730 2024-03-25T19:52:48.586 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.586+0000 7f4d29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:48.586 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.586+0000 7f4d29ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:48.586 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.586+0000 7f4d29ffb640 10 monclient: dump: 2024-03-25T19:52:48.587 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:48.587 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:48.587 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:48.587 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:48.587 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:48.587 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:48.587 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:48.588 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:48.588 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:48.588 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:48.588 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.586+0000 7f4d2b7fe640 4 set_mon_vals no callback set 2024-03-25T19:52:48.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.587+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f4d300039f0 con 0x7f4d340b1730 2024-03-25T19:52:48.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.587+0000 7f4d38d02640 10 monclient: get_auth_request con 0x7f4d1806df50 auth_method 0 2024-03-25T19:52:48.589 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.588+0000 7f4d29ffb640 10 client.4728.objecter ms_handle_connect 0x7f4d1806df50 2024-03-25T19:52:48.591 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.591+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3447814714 0 0) 0x7f4d30053610 con 0x7f4d340b1730 2024-03-25T19:52:48.591 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.591+0000 7f4d29ffb640 10 client.4728.objecter ms_dispatch 0x7f4d34000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:48.591 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.591+0000 7f4d29ffb640 3 client.4728.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:48.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.591+0000 7f4d29ffb640 3 client.4728.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:48.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.591+0000 7f4d29ffb640 20 client.4728.objecter dump_active .. 0 homeless 2024-03-25T19:52:48.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.591+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d3004ef50 con 0x7f4d340b1730 2024-03-25T19:52:48.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.591+0000 7f4d29ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:48.592 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.591+0000 7f4d29ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:48.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.809+0000 7f4d3b767640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:48.809 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.809+0000 7f4d3b767640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:48.809 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.809+0000 7f4d3b767640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:52:48.809 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.809+0000 7f4d3b767640 1 -- 172.21.15.81:0/2771465667 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d341d7690 con 0x7f4d340b1730 2024-03-25T19:52:48.855 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.855+0000 7f4d29ffb640 1 -- 172.21.15.81:0/2771465667 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3235219182) 0x7f4d30051050 con 0x7f4d340b1730 2024-03-25T19:52:48.855 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.855+0000 7f4d29ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:48.855 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.855+0000 7f4d29ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:48.855 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:52:48.856 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":11,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:52:03.690842+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:52:03.690842+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:48.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.857+0000 7f4d3b767640 10 librados: watch_flush enter 2024-03-25T19:52:48.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.857+0000 7f4d3b767640 10 librados: watch_flush exit 2024-03-25T19:52:48.858 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.857+0000 7f4d3b767640 20 client.4728.objecter shutdown clearing up homeless session... 2024-03-25T19:52:48.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.857+0000 7f4d3b767640 10 client.4728.objecter successfully canceled tick 2024-03-25T19:52:48.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.858+0000 7f4d3b767640 1 -- 172.21.15.81:0/2771465667 >> v1:172.21.15.98:6800/3980829775 conn(0x7f4d1806df50 legacy=0x7f4d18070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:48.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.858+0000 7f4d3b767640 10 monclient: shutdown 2024-03-25T19:52:48.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.858+0000 7f4d3b767640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:48.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.858+0000 7f4d3b767640 1 -- 172.21.15.81:0/2771465667 >> v1:172.21.15.81:6789/0 conn(0x7f4d340b1730 legacy=0x7f4d341405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:48.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.858+0000 7f4d3b767640 1 -- 172.21.15.81:0/2771465667 shutdown_connections 2024-03-25T19:52:48.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.858+0000 7f4d3b767640 1 -- 172.21.15.81:0/2771465667 >> 172.21.15.81:0/2771465667 conn(0x7f4d340617a0 msgr2=0x7f4d340b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:48.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.858+0000 7f4d3b767640 1 -- 172.21.15.81:0/2771465667 shutdown_connections 2024-03-25T19:52:48.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.858+0000 7f4d3b767640 1 -- 172.21.15.81:0/2771465667 wait complete. 2024-03-25T19:52:48.860 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:48.858+0000 7f4d3b767640 1 librados: shutdown 2024-03-25T19:52:48.870 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:48.954 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:11:11.596 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:48.962 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:11:11.604 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:48.997 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:11:11.639 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:49.024 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:11:11.666 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:49.031 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:11:11.673 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:49.043 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:11.677 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:49.046 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:11.681 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:49.068 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:11:11.710 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:49.079 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:11.714 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:55.677 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:55.793 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.793+0000 7fa68624e640 10 monclient: get_monmap_and_config 2024-03-25T19:52:55.794 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.793+0000 7fa68624e640 10 monclient: build_initial_monmap 2024-03-25T19:52:55.794 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.793+0000 7fa68624e640 10 monclient: monmap: 2024-03-25T19:52:55.794 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:52:55.794 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:55.794 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:52:55.795004+0000 2024-03-25T19:52:55.794 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:52:55.795004+0000 2024-03-25T19:52:55.794 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:55.794 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:55.794 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:52:55.794 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:52:55.794 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:52:55.794 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:55.795 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.793+0000 7fa68624e640 1 Processor -- start 2024-03-25T19:52:55.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.793+0000 7fa68624e640 1 -- start start 2024-03-25T19:52:55.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.793+0000 7fa68624e640 10 monclient: init 2024-03-25T19:52:55.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.794+0000 7fa68624e640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:55.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.794+0000 7fa68624e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:52:55.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.794+0000 7fa68624e640 10 monclient(hunting): picked mon.noname-c con 0x7fa6800b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:52:55.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.794+0000 7fa68624e640 10 monclient(hunting): picked mon.noname-a con 0x7fa6800b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:52:55.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.794+0000 7fa68624e640 10 monclient(hunting): picked mon.noname-b con 0x7fa68014be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:52:55.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.794+0000 7fa68624e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6800af940 con 0x7fa6800b1730 2024-03-25T19:52:55.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.794+0000 7fa68624e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6800b0a70 con 0x7fa6800b4d60 2024-03-25T19:52:55.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.794+0000 7fa68624e640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6800b0d50 con 0x7fa68014be90 2024-03-25T19:52:55.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.794+0000 7fa68624e640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:55.796 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.794+0000 7fa68624e640 10 monclient(hunting): authenticate will time out at 4610.877469s 2024-03-25T19:52:55.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.802+0000 7fa67ffff640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7fa68014be90 0x7fa68014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:38958/0 (socket says 172.21.15.81:38958) 2024-03-25T19:52:55.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.802+0000 7fa67ffff640 1 -- 172.21.15.81:0/159750405 learned_addr learned my addr 172.21.15.81:0/159750405 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:52:55.817 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.816+0000 7fa67e7fc640 1 -- 172.21.15.81:0/159750405 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870567259 0 0) 0x7fa6800b0d50 con 0x7fa68014be90 2024-03-25T19:52:55.817 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.817+0000 7fa67e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:55.817 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.817+0000 7fa67e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:55.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.817+0000 7fa67e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:55.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.817+0000 7fa67e7fc640 10 monclient(hunting): my global_id is 4688 2024-03-25T19:52:55.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.817+0000 7fa67e7fc640 1 -- 172.21.15.81:0/159750405 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa668003520 con 0x7fa68014be90 2024-03-25T19:52:55.823 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.823+0000 7fa67e7fc640 1 -- 172.21.15.81:0/159750405 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948543645 0 0) 0x7fa6800b0a70 con 0x7fa6800b4d60 2024-03-25T19:52:55.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.823+0000 7fa67e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:55.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.823+0000 7fa67e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:55.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.823+0000 7fa67e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:55.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.823+0000 7fa67e7fc640 10 monclient(hunting): my global_id is 4708 2024-03-25T19:52:55.824 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.823+0000 7fa67e7fc640 1 -- 172.21.15.81:0/159750405 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6800b0d50 con 0x7fa6800b4d60 2024-03-25T19:52:55.829 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.827+0000 7fa67e7fc640 1 -- 172.21.15.81:0/159750405 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2164989866 0 0) 0x7fa668003520 con 0x7fa68014be90 2024-03-25T19:52:55.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.827+0000 7fa67e7fc640 1 -- 172.21.15.81:0/159750405 >> v1:172.21.15.98:6789/0 conn(0x7fa6800b4d60 legacy=0x7fa68013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:55.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.827+0000 7fa67e7fc640 1 -- 172.21.15.81:0/159750405 >> v1:172.21.15.81:6789/0 conn(0x7fa6800b1730 legacy=0x7fa6800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:55.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.827+0000 7fa67e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:52:55.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.827+0000 7fa67e7fc640 1 monclient: found mon.noname-b 2024-03-25T19:52:55.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.827+0000 7fa67e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:55.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.827+0000 7fa67e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.81:6790/0 2024-03-25T19:52:55.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.827+0000 7fa67e7fc640 1 -- 172.21.15.81:0/159750405 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6800b1030 con 0x7fa68014be90 2024-03-25T19:52:55.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.827+0000 7fa67e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:52:55.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.827+0000 7fa67e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:52:55.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.827+0000 7fa67e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:55.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.827+0000 7fa67e7fc640 1 -- 172.21.15.81:0/159750405 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa678001a10 con 0x7fa68014be90 2024-03-25T19:52:55.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.827+0000 7fa67e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:55.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.827+0000 7fa67e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:55.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.827+0000 7fa67e7fc640 10 monclient: dump: 2024-03-25T19:52:55.831 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:55.831 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:55.831 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:55.831 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:55.831 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:55.831 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:55.831 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:55.831 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:55.831 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:55.831 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:55.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.827+0000 7fa68624e640 5 monclient: authenticate success, global_id 4688 2024-03-25T19:52:55.831 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.827+0000 7fa68624e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:55.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.845+0000 7fa67e7fc640 1 -- 172.21.15.81:0/159750405 <== mon.2 v1:172.21.15.81:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa678001b70 con 0x7fa68014be90 2024-03-25T19:52:55.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.845+0000 7fa67e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:55.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.845+0000 7fa67e7fc640 1 -- 172.21.15.81:0/159750405 <== mon.2 v1:172.21.15.81:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa678001f80 con 0x7fa68014be90 2024-03-25T19:52:55.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.845+0000 7fa68624e640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:55.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.845+0000 7fa68624e640 4 set_mon_vals no callback set 2024-03-25T19:52:55.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.845+0000 7fa67e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:55.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.845+0000 7fa67e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:52:55.846 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.845+0000 7fa67e7fc640 10 monclient: dump: 2024-03-25T19:52:55.846 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:55.846 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:55.846 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:55.846 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:55.846 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:55.846 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:55.847 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:55.847 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:55.847 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:55.847 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:55.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.846+0000 7fa68624e640 10 monclient: shutdown 2024-03-25T19:52:55.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.846+0000 7fa68624e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:55.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.846+0000 7fa68624e640 1 -- 172.21.15.81:0/159750405 >> v1:172.21.15.81:6790/0 conn(0x7fa68014be90 legacy=0x7fa68014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:55.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.846+0000 7fa68624e640 1 -- 172.21.15.81:0/159750405 shutdown_connections 2024-03-25T19:52:55.849 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.846+0000 7fa68624e640 1 -- 172.21.15.81:0/159750405 >> 172.21.15.81:0/159750405 conn(0x7fa6800617a0 msgr2=0x7fa680061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:55.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.846+0000 7fa68624e640 1 -- 172.21.15.81:0/159750405 shutdown_connections 2024-03-25T19:52:55.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.846+0000 7fa68624e640 1 -- 172.21.15.81:0/159750405 wait complete. 2024-03-25T19:52:55.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.847+0000 7fa68624e640 10 monclient: build_initial_monmap 2024-03-25T19:52:55.850 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.847+0000 7fa68624e640 10 monclient: monmap: 2024-03-25T19:52:55.850 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:52:55.850 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:55.850 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:52:55.850 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:52:55.850 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:55.850 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:52:55.850 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:52:55.850 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:52:55.850 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:52:55.850 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:55.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.847+0000 7fa68624e640 1 librados: starting msgr at 2024-03-25T19:52:55.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.847+0000 7fa68624e640 1 librados: starting objecter 2024-03-25T19:52:55.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.847+0000 7fa68624e640 1 Processor -- start 2024-03-25T19:52:55.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.847+0000 7fa68624e640 1 -- start start 2024-03-25T19:52:55.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.847+0000 7fa68624e640 1 librados: setting wanted keys 2024-03-25T19:52:55.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.847+0000 7fa68624e640 1 librados: calling monclient init 2024-03-25T19:52:55.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.847+0000 7fa68624e640 10 monclient: init 2024-03-25T19:52:55.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.848+0000 7fa68624e640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:55.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.848+0000 7fa68624e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:52:55.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.848+0000 7fa68624e640 10 monclient(hunting): picked mon.noname-a con 0x7fa6800b1730 addr v1:172.21.15.81:6789/0 2024-03-25T19:52:55.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.848+0000 7fa68624e640 10 monclient(hunting): picked mon.noname-b con 0x7fa6800b4d60 addr v1:172.21.15.98:6789/0 2024-03-25T19:52:55.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.848+0000 7fa68624e640 10 monclient(hunting): picked mon.noname-c con 0x7fa68014be90 addr v1:172.21.15.81:6790/0 2024-03-25T19:52:55.851 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.848+0000 7fa68624e640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa680140740 con 0x7fa6800b1730 2024-03-25T19:52:55.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.848+0000 7fa68624e640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa680135810 con 0x7fa6800b4d60 2024-03-25T19:52:55.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.848+0000 7fa68624e640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa680140bd0 con 0x7fa68014be90 2024-03-25T19:52:55.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.849+0000 7fa68624e640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:55.852 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.849+0000 7fa68624e640 10 monclient(hunting): authenticate will time out at 4610.931847s 2024-03-25T19:52:55.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.856+0000 7fa67effd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fa6800b4d60 0x7fa6801349f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:57178/0 (socket says 172.21.15.81:57178) 2024-03-25T19:52:55.856 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.856+0000 7fa67effd640 1 -- 172.21.15.81:0/1041973508 learned_addr learned my addr 172.21.15.81:0/1041973508 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:52:55.857 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.856+0000 7fa667fff640 10 client.?.objecter ms_handle_connect 0x7fa6800b4d60 2024-03-25T19:52:55.857 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.856+0000 7fa667fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:55.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.858+0000 7fa667fff640 10 client.?.objecter ms_handle_connect 0x7fa68014be90 2024-03-25T19:52:55.859 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.858+0000 7fa667fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:55.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.861+0000 7fa667fff640 10 client.?.objecter ms_handle_connect 0x7fa6800b1730 2024-03-25T19:52:55.862 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.861+0000 7fa667fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:55.868 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.867+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836825775 0 0) 0x7fa680135810 con 0x7fa6800b4d60 2024-03-25T19:52:55.868 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.867+0000 7fa667fff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:55.868 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.867+0000 7fa667fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:55.868 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.867+0000 7fa667fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:55.868 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.867+0000 7fa667fff640 10 monclient(hunting): my global_id is 4711 2024-03-25T19:52:55.868 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.867+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa66c004eb0 con 0x7fa6800b4d60 2024-03-25T19:52:55.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.869+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872591111 0 0) 0x7fa680140bd0 con 0x7fa68014be90 2024-03-25T19:52:55.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.869+0000 7fa667fff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:55.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.869+0000 7fa667fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:55.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.869+0000 7fa667fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:55.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.869+0000 7fa667fff640 10 monclient(hunting): my global_id is 4691 2024-03-25T19:52:55.871 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.869+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa680135810 con 0x7fa68014be90 2024-03-25T19:52:55.871 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.869+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225137212 0 0) 0x7fa680140740 con 0x7fa6800b1730 2024-03-25T19:52:55.871 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.869+0000 7fa667fff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:55.871 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.869+0000 7fa667fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:55.871 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.869+0000 7fa667fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:55.871 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.869+0000 7fa667fff640 10 monclient(hunting): my global_id is 4734 2024-03-25T19:52:55.871 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.869+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa680140bd0 con 0x7fa6800b1730 2024-03-25T19:52:55.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.876+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3289487581 0 0) 0x7fa66c004eb0 con 0x7fa6800b4d60 2024-03-25T19:52:55.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.876+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa680140740 con 0x7fa6800b4d60 2024-03-25T19:52:55.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.876+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa6740019d0 con 0x7fa6800b4d60 2024-03-25T19:52:55.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.876+0000 7fa667fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:55.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.876+0000 7fa667fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:55.877 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.876+0000 7fa667fff640 10 monclient(hunting): dump: 2024-03-25T19:52:55.877 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:55.877 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:55.877 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:55.877 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:55.877 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:55.877 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:55.877 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:55.877 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:55.878 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:55.878 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:55.878 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.876+0000 7fa667fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:55.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3894183091 0 0) 0x7fa680140740 con 0x7fa6800b4d60 2024-03-25T19:52:55.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 >> v1:172.21.15.81:6790/0 conn(0x7fa68014be90 legacy=0x7fa680135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:55.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 >> v1:172.21.15.81:6789/0 conn(0x7fa6800b1730 legacy=0x7fa6801d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:55.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa667fff640 10 monclient: _finish_hunting 0 2024-03-25T19:52:55.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa667fff640 1 monclient: found mon.b 2024-03-25T19:52:55.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa667fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:55.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa667fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:55.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa680140380 con 0x7fa6800b4d60 2024-03-25T19:52:55.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa667fff640 10 monclient: _finish_auth 0 2024-03-25T19:52:55.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa667fff640 10 monclient: _check_auth_tickets 2024-03-25T19:52:55.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa667fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:55.883 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa68624e640 5 monclient: authenticate success, global_id 4711 2024-03-25T19:52:55.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa68624e640 10 monclient: _renew_subs 2024-03-25T19:52:55.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa68624e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:55.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa68624e640 1 -- 172.21.15.81:0/1041973508 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa680145710 con 0x7fa6800b4d60 2024-03-25T19:52:55.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa68624e640 10 client.4711.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:55.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa68624e640 10 monclient: _renew_subs 2024-03-25T19:52:55.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa68624e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:55.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa68624e640 1 -- 172.21.15.81:0/1041973508 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa680144ec0 con 0x7fa6800b4d60 2024-03-25T19:52:55.884 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.882+0000 7fa68624e640 1 librados: init done 2024-03-25T19:52:55.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.883+0000 7fa68624e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:55.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.883+0000 7fa68624e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:55.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.883+0000 7fa68624e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:55.885 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:55.883+0000 7fa68624e640 1 -- 172.21.15.81:0/1041973508 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6800afbd0 con 0x7fa6800b4d60 2024-03-25T19:52:56.161 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.161+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6740035f0 con 0x7fa6800b4d60 2024-03-25T19:52:56.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.161+0000 7fa667fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:56.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.161+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fa674003a00 con 0x7fa6800b4d60 2024-03-25T19:52:56.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.161+0000 7fa667fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:56.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.161+0000 7fa667fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:56.162 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.161+0000 7fa667fff640 10 monclient: dump: 2024-03-25T19:52:56.162 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:52:56.162 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:52:56.162 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:56.162 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:52:56.162 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:56.162 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:52:56.162 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:52:56.162 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:52:56.162 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:52:56.163 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:52:56.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.161+0000 7fa67dffb640 4 set_mon_vals no callback set 2024-03-25T19:52:56.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.162+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fa674003c20 con 0x7fa6800b4d60 2024-03-25T19:52:56.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.162+0000 7fa67f7fe640 10 monclient: get_auth_request con 0x7fa66c06ddf0 auth_method 0 2024-03-25T19:52:56.163 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.163+0000 7fa667fff640 10 client.4711.objecter ms_handle_connect 0x7fa66c06ddf0 2024-03-25T19:52:56.168 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.168+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3447814714 0 0) 0x7fa6740547d0 con 0x7fa6800b4d60 2024-03-25T19:52:56.168 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.168+0000 7fa667fff640 10 client.4711.objecter ms_dispatch 0x7fa680000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:52:56.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.168+0000 7fa667fff640 3 client.4711.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:52:56.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.168+0000 7fa667fff640 3 client.4711.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:52:56.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.168+0000 7fa667fff640 20 client.4711.objecter dump_active .. 0 homeless 2024-03-25T19:52:56.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.168+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6740502e0 con 0x7fa6800b4d60 2024-03-25T19:52:56.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.168+0000 7fa667fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:56.169 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.168+0000 7fa667fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:56.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.415+0000 7fa68624e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:56.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.415+0000 7fa68624e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:56.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.415+0000 7fa68624e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:52:56.416 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.415+0000 7fa68624e640 1 -- 172.21.15.81:0/1041973508 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa680140510 con 0x7fa6800b4d60 2024-03-25T19:52:56.447 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.447+0000 7fa667fff640 1 -- 172.21.15.81:0/1041973508 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3235219182) 0x7fa674052210 con 0x7fa6800b4d60 2024-03-25T19:52:56.448 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.447+0000 7fa667fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:56.448 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.447+0000 7fa667fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:56.448 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:52:56.448 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":11,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:52:03.690842+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:52:03.690842+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:56.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.449+0000 7fa68624e640 10 librados: watch_flush enter 2024-03-25T19:52:56.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.449+0000 7fa68624e640 10 librados: watch_flush exit 2024-03-25T19:52:56.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.449+0000 7fa68624e640 20 client.4711.objecter shutdown clearing up homeless session... 2024-03-25T19:52:56.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.449+0000 7fa68624e640 10 client.4711.objecter successfully canceled tick 2024-03-25T19:52:56.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.450+0000 7fa68624e640 1 -- 172.21.15.81:0/1041973508 >> v1:172.21.15.98:6800/3980829775 conn(0x7fa66c06ddf0 legacy=0x7fa66c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:56.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.450+0000 7fa68624e640 10 monclient: shutdown 2024-03-25T19:52:56.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.450+0000 7fa68624e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:56.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.450+0000 7fa68624e640 1 -- 172.21.15.81:0/1041973508 >> v1:172.21.15.98:6789/0 conn(0x7fa6800b4d60 legacy=0x7fa6801349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:56.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.450+0000 7fa68624e640 1 -- 172.21.15.81:0/1041973508 shutdown_connections 2024-03-25T19:52:56.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.450+0000 7fa68624e640 1 -- 172.21.15.81:0/1041973508 >> 172.21.15.81:0/1041973508 conn(0x7fa6800617a0 msgr2=0x7fa680138590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:56.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.450+0000 7fa68624e640 1 -- 172.21.15.81:0/1041973508 shutdown_connections 2024-03-25T19:52:56.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.450+0000 7fa68624e640 1 -- 172.21.15.81:0/1041973508 wait complete. 2024-03-25T19:52:56.451 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:52:56.450+0000 7fa68624e640 1 librados: shutdown 2024-03-25T19:52:56.462 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:59.583 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:11:22.225 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:59.589 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:11:22.231 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:59.595 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:11:22.237 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:59.602 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:11:22.243 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:59.602 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:22.236 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:59.604 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:22.239 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:59.613 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:11:22.255 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:59.625 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:22.260 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:59.626 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:11:22.268 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:03.267 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:53:03.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.381+0000 7f171c464640 10 monclient: get_monmap_and_config 2024-03-25T19:53:03.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.381+0000 7f171c464640 10 monclient: build_initial_monmap 2024-03-25T19:53:03.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.381+0000 7f171c464640 10 monclient: monmap: 2024-03-25T19:53:03.380 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:53:03.381 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:03.381 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:53:03.381947+0000 2024-03-25T19:53:03.381 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:53:03.381947+0000 2024-03-25T19:53:03.381 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:53:03.381 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:03.381 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:53:03.381 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:53:03.381 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:53:03.381 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:03.382 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.381+0000 7f171c464640 1 Processor -- start 2024-03-25T19:53:03.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.382+0000 7f171c464640 1 -- start start 2024-03-25T19:53:03.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.382+0000 7f171c464640 10 monclient: init 2024-03-25T19:53:03.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.382+0000 7f171c464640 10 monclient: _reopen_session rank -1 2024-03-25T19:53:03.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.382+0000 7f171c464640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:53:03.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.382+0000 7f171c464640 10 monclient(hunting): picked mon.noname-a con 0x7f171406ea40 addr v1:172.21.15.81:6789/0 2024-03-25T19:53:03.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.382+0000 7f171c464640 10 monclient(hunting): picked mon.noname-c con 0x7f1714071800 addr v1:172.21.15.98:6789/0 2024-03-25T19:53:03.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.383+0000 7f171c464640 10 monclient(hunting): picked mon.noname-b con 0x7f1714071c40 addr v1:172.21.15.81:6790/0 2024-03-25T19:53:03.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.383+0000 7f171c464640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171406f1f0 con 0x7f171406ea40 2024-03-25T19:53:03.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.383+0000 7f171c464640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1714070d00 con 0x7f1714071800 2024-03-25T19:53:03.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.383+0000 7f171c464640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17140720b0 con 0x7f1714071c40 2024-03-25T19:53:03.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.383+0000 7f171c464640 10 monclient(hunting): _renew_subs 2024-03-25T19:53:03.383 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.383+0000 7f171c464640 10 monclient(hunting): authenticate will time out at 4618.465115s 2024-03-25T19:53:03.384 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.383+0000 7f17199ff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f1714071800 0x7f1714147070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:33770/0 (socket says 172.21.15.81:33770) 2024-03-25T19:53:03.384 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.383+0000 7f17199ff640 1 -- 172.21.15.81:0/3262886662 learned_addr learned my addr 172.21.15.81:0/3262886662 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:53:03.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.457+0000 7f17191fe640 1 -- 172.21.15.81:0/3262886662 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208634128 0 0) 0x7f171406f1f0 con 0x7f171406ea40 2024-03-25T19:53:03.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.457+0000 7f17191fe640 10 monclient(hunting): _negotiate 2024-03-25T19:53:03.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.457+0000 7f17191fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:03.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.457+0000 7f17191fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:03.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.457+0000 7f17191fe640 10 monclient(hunting): my global_id is 4737 2024-03-25T19:53:03.457 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.457+0000 7f17191fe640 1 -- 172.21.15.81:0/3262886662 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1700003520 con 0x7f171406ea40 2024-03-25T19:53:03.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.477+0000 7f17191fe640 1 -- 172.21.15.81:0/3262886662 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1248446114 0 0) 0x7f1700003520 con 0x7f171406ea40 2024-03-25T19:53:03.477 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.477+0000 7f17191fe640 1 -- 172.21.15.81:0/3262886662 >> v1:172.21.15.81:6790/0 conn(0x7f1714071c40 legacy=0x7f171414a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:03.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.477+0000 7f17191fe640 1 -- 172.21.15.81:0/3262886662 >> v1:172.21.15.98:6789/0 conn(0x7f1714071800 legacy=0x7f1714147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:03.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.477+0000 7f17191fe640 10 monclient: _finish_hunting 0 2024-03-25T19:53:03.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.477+0000 7f17191fe640 1 monclient: found mon.noname-a 2024-03-25T19:53:03.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.477+0000 7f17191fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:53:03.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.478+0000 7f17191fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:53:03.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.478+0000 7f17191fe640 1 -- 172.21.15.81:0/3262886662 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1714072780 con 0x7f171406ea40 2024-03-25T19:53:03.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.478+0000 7f17191fe640 10 monclient: _finish_auth 0 2024-03-25T19:53:03.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.478+0000 7f17191fe640 10 monclient: _check_auth_tickets 2024-03-25T19:53:03.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.478+0000 7f17191fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:53:03.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.478+0000 7f17191fe640 1 -- 172.21.15.81:0/3262886662 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1710001690 con 0x7f171406ea40 2024-03-25T19:53:03.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.478+0000 7f17191fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:03.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.478+0000 7f17191fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:53:03.478 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.478+0000 7f17191fe640 10 monclient: dump: 2024-03-25T19:53:03.479 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:53:03.479 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:03.479 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:03.479 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:03.479 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:03.479 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:03.479 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:53:03.479 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:53:03.479 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:53:03.479 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:03.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.478+0000 7f171c464640 5 monclient: authenticate success, global_id 4737 2024-03-25T19:53:03.479 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.478+0000 7f171c464640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:53:03.499 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.500+0000 7f17191fe640 1 -- 172.21.15.81:0/3262886662 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17100017f0 con 0x7f171406ea40 2024-03-25T19:53:03.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.500+0000 7f17191fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:53:03.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.500+0000 7f17191fe640 1 -- 172.21.15.81:0/3262886662 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f1710001c00 con 0x7f171406ea40 2024-03-25T19:53:03.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.500+0000 7f17191fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:03.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.500+0000 7f17191fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:53:03.500 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.500+0000 7f17191fe640 10 monclient: dump: 2024-03-25T19:53:03.500 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:53:03.500 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:03.500 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:03.500 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:03.500 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:03.500 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:03.500 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:53:03.500 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:53:03.500 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:53:03.501 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:03.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.500+0000 7f171c464640 10 monclient: get_monmap_and_config success 2024-03-25T19:53:03.501 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.500+0000 7f171c464640 4 set_mon_vals no callback set 2024-03-25T19:53:03.502 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.501+0000 7f171c464640 10 monclient: shutdown 2024-03-25T19:53:03.502 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.501+0000 7f171c464640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:53:03.502 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.501+0000 7f171c464640 1 -- 172.21.15.81:0/3262886662 >> v1:172.21.15.81:6789/0 conn(0x7f171406ea40 legacy=0x7f171413ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:03.502 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.501+0000 7f171c464640 1 -- 172.21.15.81:0/3262886662 shutdown_connections 2024-03-25T19:53:03.502 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.501+0000 7f171c464640 1 -- 172.21.15.81:0/3262886662 >> 172.21.15.81:0/3262886662 conn(0x7f171406b630 msgr2=0x7f17140b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:53:03.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.501+0000 7f171c464640 1 -- 172.21.15.81:0/3262886662 shutdown_connections 2024-03-25T19:53:03.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.501+0000 7f171c464640 1 -- 172.21.15.81:0/3262886662 wait complete. 2024-03-25T19:53:03.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.502+0000 7f171c464640 10 monclient: build_initial_monmap 2024-03-25T19:53:03.503 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.502+0000 7f171c464640 10 monclient: monmap: 2024-03-25T19:53:03.503 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:53:03.503 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:53:03.503 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:53:03.503 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:53:03.503 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:53:03.503 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:53:03.503 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:53:03.503 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:53:03.503 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:53:03.503 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:03.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.502+0000 7f171c464640 1 librados: starting msgr at 2024-03-25T19:53:03.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.502+0000 7f171c464640 1 librados: starting objecter 2024-03-25T19:53:03.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.503+0000 7f171c464640 1 Processor -- start 2024-03-25T19:53:03.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.503+0000 7f171c464640 1 -- start start 2024-03-25T19:53:03.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.503+0000 7f171c464640 1 librados: setting wanted keys 2024-03-25T19:53:03.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.503+0000 7f171c464640 1 librados: calling monclient init 2024-03-25T19:53:03.504 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.503+0000 7f171c464640 10 monclient: init 2024-03-25T19:53:03.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.503+0000 7f171c464640 10 monclient: _reopen_session rank -1 2024-03-25T19:53:03.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.503+0000 7f171c464640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:53:03.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.503+0000 7f171c464640 10 monclient(hunting): picked mon.noname-c con 0x7f171406ea40 addr v1:172.21.15.81:6790/0 2024-03-25T19:53:03.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.503+0000 7f171c464640 10 monclient(hunting): picked mon.noname-a con 0x7f1714071800 addr v1:172.21.15.81:6789/0 2024-03-25T19:53:03.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.503+0000 7f171c464640 10 monclient(hunting): picked mon.noname-b con 0x7f1714071c40 addr v1:172.21.15.98:6789/0 2024-03-25T19:53:03.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.503+0000 7f171c464640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1714131e30 con 0x7f1714071800 2024-03-25T19:53:03.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.503+0000 7f171c464640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1714133900 con 0x7f1714071c40 2024-03-25T19:53:03.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.504+0000 7f171c464640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17141d4fe0 con 0x7f171406ea40 2024-03-25T19:53:03.505 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.504+0000 7f171c464640 10 monclient(hunting): _renew_subs 2024-03-25T19:53:03.506 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.504+0000 7f171c464640 10 monclient(hunting): authenticate will time out at 4618.585871s 2024-03-25T19:53:03.510 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.511+0000 7f17199ff640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7f1714071800 0x7f17141d7460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:45674/0 (socket says 172.21.15.81:45674) 2024-03-25T19:53:03.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.511+0000 7f17199ff640 1 -- 172.21.15.81:0/2656757736 learned_addr learned my addr 172.21.15.81:0/2656757736 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:53:03.518 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.518+0000 7f170affd640 10 client.?.objecter ms_handle_connect 0x7f1714071800 2024-03-25T19:53:03.518 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.518+0000 7f170affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:03.530 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.530+0000 7f170affd640 10 client.?.objecter ms_handle_connect 0x7f1714071c40 2024-03-25T19:53:03.530 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.530+0000 7f170affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:03.534 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.534+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112052339 0 0) 0x7f1714131e30 con 0x7f1714071800 2024-03-25T19:53:03.534 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.534+0000 7f170affd640 10 monclient(hunting): _negotiate 2024-03-25T19:53:03.534 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.534+0000 7f170affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:03.534 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.534+0000 7f170affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:03.534 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.534+0000 7f170affd640 10 monclient(hunting): my global_id is 4740 2024-03-25T19:53:03.534 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.534+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f8004eb0 con 0x7f1714071800 2024-03-25T19:53:03.542 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.543+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059329203 0 0) 0x7f1714133900 con 0x7f1714071c40 2024-03-25T19:53:03.543 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.543+0000 7f170affd640 10 monclient(hunting): _negotiate 2024-03-25T19:53:03.543 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.543+0000 7f170affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:03.543 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.543+0000 7f170affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:03.543 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.543+0000 7f170affd640 10 monclient(hunting): my global_id is 4714 2024-03-25T19:53:03.543 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.543+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1714131e30 con 0x7f1714071c40 2024-03-25T19:53:03.545 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.546+0000 7f170affd640 10 client.?.objecter ms_handle_connect 0x7f171406ea40 2024-03-25T19:53:03.545 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.546+0000 7f170affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:03.554 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.555+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264937551 0 0) 0x7f17141d4fe0 con 0x7f171406ea40 2024-03-25T19:53:03.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.555+0000 7f170affd640 10 monclient(hunting): _negotiate 2024-03-25T19:53:03.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.555+0000 7f170affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:03.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.555+0000 7f170affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:03.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.555+0000 7f170affd640 10 monclient(hunting): my global_id is 4697 2024-03-25T19:53:03.555 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.555+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1714133900 con 0x7f171406ea40 2024-03-25T19:53:03.557 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.558+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2501045745 0 0) 0x7f1714131e30 con 0x7f1714071c40 2024-03-25T19:53:03.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.558+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17141d4fe0 con 0x7f1714071c40 2024-03-25T19:53:03.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.558+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f170c001b00 con 0x7f1714071c40 2024-03-25T19:53:03.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.558+0000 7f170affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:03.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.558+0000 7f170affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:53:03.558 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.558+0000 7f170affd640 10 monclient(hunting): dump: 2024-03-25T19:53:03.558 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:53:03.558 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:03.558 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:03.558 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:03.558 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:03.558 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:03.558 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:53:03.559 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:53:03.559 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:53:03.559 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:03.559 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.558+0000 7f170affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:53:03.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.566+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4182725860 0 0) 0x7f16f8004eb0 con 0x7f1714071800 2024-03-25T19:53:03.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.566+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1714131e30 con 0x7f1714071800 2024-03-25T19:53:03.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.566+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f17040018b0 con 0x7f1714071800 2024-03-25T19:53:03.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.566+0000 7f170affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:03.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.566+0000 7f170affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:53:03.566 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.566+0000 7f170affd640 10 monclient(hunting): dump: 2024-03-25T19:53:03.567 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:53:03.567 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:03.567 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:03.567 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:03.567 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:03.567 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:03.567 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:53:03.567 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:53:03.567 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:53:03.567 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:03.573 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.574+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1491759334 0 0) 0x7f17141d4fe0 con 0x7f1714071c40 2024-03-25T19:53:03.574 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.574+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 >> v1:172.21.15.81:6790/0 conn(0x7f171406ea40 legacy=0x7f17140ad180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:03.574 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.574+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 >> v1:172.21.15.81:6789/0 conn(0x7f1714071800 legacy=0x7f17141d7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:03.574 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.574+0000 7f170affd640 10 monclient: _finish_hunting 0 2024-03-25T19:53:03.574 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.574+0000 7f170affd640 1 monclient: found mon.b 2024-03-25T19:53:03.574 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.574+0000 7f170affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:53:03.574 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.574+0000 7f170affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:53:03.574 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.574+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17141372f0 con 0x7f1714071c40 2024-03-25T19:53:03.574 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.574+0000 7f170affd640 10 monclient: _finish_auth 0 2024-03-25T19:53:03.574 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.574+0000 7f170affd640 10 monclient: _check_auth_tickets 2024-03-25T19:53:03.574 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.574+0000 7f170affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:53:03.575 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.575+0000 7f171c464640 5 monclient: authenticate success, global_id 4714 2024-03-25T19:53:03.575 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.575+0000 7f171c464640 10 monclient: _renew_subs 2024-03-25T19:53:03.575 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.575+0000 7f171c464640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:53:03.576 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.575+0000 7f171c464640 1 -- 172.21.15.81:0/2656757736 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17141375a0 con 0x7f1714071c40 2024-03-25T19:53:03.576 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.575+0000 7f171c464640 10 client.4714.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:53:03.576 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.575+0000 7f171c464640 10 monclient: _renew_subs 2024-03-25T19:53:03.576 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.575+0000 7f171c464640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:53:03.576 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.575+0000 7f171c464640 1 -- 172.21.15.81:0/2656757736 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17141324a0 con 0x7f1714071c40 2024-03-25T19:53:03.576 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.575+0000 7f171c464640 1 librados: init done 2024-03-25T19:53:03.576 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.576+0000 7f171c464640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:03.576 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.576+0000 7f171c464640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:03.576 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.576+0000 7f171c464640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:53:03.576 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.576+0000 7f171c464640 1 -- 172.21.15.81:0/2656757736 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17141327f0 con 0x7f1714071c40 2024-03-25T19:53:03.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.870+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f170c001ff0 con 0x7f1714071c40 2024-03-25T19:53:03.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.870+0000 7f170affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:53:03.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.870+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f170c003970 con 0x7f1714071c40 2024-03-25T19:53:03.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.870+0000 7f170affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:03.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.870+0000 7f170affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:53:03.870 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.870+0000 7f170affd640 10 monclient: dump: 2024-03-25T19:53:03.870 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:53:03.870 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:03.871 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:03.871 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:03.871 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:03.871 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:03.871 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:53:03.871 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:53:03.871 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:53:03.871 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:03.871 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.870+0000 7f17189fd640 4 set_mon_vals no callback set 2024-03-25T19:53:03.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.871+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f170c003b50 con 0x7f1714071c40 2024-03-25T19:53:03.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.871+0000 7f171a200640 10 monclient: get_auth_request con 0x7f16f806df70 auth_method 0 2024-03-25T19:53:03.872 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.871+0000 7f170affd640 10 client.4714.objecter ms_handle_connect 0x7f16f806df70 2024-03-25T19:53:03.875 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.876+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3447814714 0 0) 0x7f170c0535c0 con 0x7f1714071c40 2024-03-25T19:53:03.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.876+0000 7f170affd640 10 client.4714.objecter ms_dispatch 0x7f1714000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:53:03.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.876+0000 7f170affd640 3 client.4714.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:53:03.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.876+0000 7f170affd640 3 client.4714.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:53:03.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.876+0000 7f170affd640 20 client.4714.objecter dump_active .. 0 homeless 2024-03-25T19:53:03.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.877+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f170c04f0d0 con 0x7f1714071c40 2024-03-25T19:53:03.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.877+0000 7f170affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:03.876 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:03.877+0000 7f170affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:53:04.130 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.130+0000 7f171c464640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:04.130 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.130+0000 7f171c464640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:04.130 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.130+0000 7f171c464640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:53:04.130 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.130+0000 7f171c464640 1 -- 172.21.15.81:0/2656757736 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1714131fe0 con 0x7f1714071c40 2024-03-25T19:53:04.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.156+0000 7f170affd640 1 -- 172.21.15.81:0/2656757736 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3235219182) 0x7f1714131fe0 con 0x7f1714071c40 2024-03-25T19:53:04.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.156+0000 7f170affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:04.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.156+0000 7f170affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:53:04.156 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:53:04.156 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":11,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:52:03.690842+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:52:03.690842+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:53:04.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.158+0000 7f171c464640 10 librados: watch_flush enter 2024-03-25T19:53:04.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.158+0000 7f171c464640 10 librados: watch_flush exit 2024-03-25T19:53:04.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.158+0000 7f171c464640 20 client.4714.objecter shutdown clearing up homeless session... 2024-03-25T19:53:04.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.158+0000 7f171c464640 10 client.4714.objecter successfully canceled tick 2024-03-25T19:53:04.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.159+0000 7f171c464640 1 -- 172.21.15.81:0/2656757736 >> v1:172.21.15.98:6800/3980829775 conn(0x7f16f806df70 legacy=0x7f16f8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:04.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.159+0000 7f171c464640 10 monclient: shutdown 2024-03-25T19:53:04.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.159+0000 7f171c464640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:53:04.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.159+0000 7f171c464640 1 -- 172.21.15.81:0/2656757736 >> v1:172.21.15.98:6789/0 conn(0x7f1714071c40 legacy=0x7f17141e1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:04.159 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.159+0000 7f171c464640 1 -- 172.21.15.81:0/2656757736 shutdown_connections 2024-03-25T19:53:04.160 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.159+0000 7f171c464640 1 -- 172.21.15.81:0/2656757736 >> 172.21.15.81:0/2656757736 conn(0x7f171406b630 msgr2=0x7f17140af690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:53:04.160 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.159+0000 7f171c464640 1 -- 172.21.15.81:0/2656757736 shutdown_connections 2024-03-25T19:53:04.160 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.159+0000 7f171c464640 1 -- 172.21.15.81:0/2656757736 wait complete. 2024-03-25T19:53:04.160 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:04.159+0000 7f171c464640 1 librados: shutdown 2024-03-25T19:53:04.171 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:53:08.978 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:11:31.620 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:09.004 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:11:31.645 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:09.051 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:11:31.693 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:09.054 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:11:31.696 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:09.061 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:31.695 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:09.073 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:31.707 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:09.073 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:11:31.715 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:09.096 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:31.731 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:09.103 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:11:31.745 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:10.977 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:53:11.092 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.092+0000 7f5106833640 10 monclient: get_monmap_and_config 2024-03-25T19:53:11.092 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.092+0000 7f5106833640 10 monclient: build_initial_monmap 2024-03-25T19:53:11.092 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.092+0000 7f5106833640 10 monclient: monmap: 2024-03-25T19:53:11.093 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:53:11.093 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:11.093 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:53:11.093738+0000 2024-03-25T19:53:11.093 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:53:11.093738+0000 2024-03-25T19:53:11.093 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:53:11.093 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:11.093 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:53:11.093 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:53:11.093 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:53:11.093 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:11.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.093+0000 7f5106833640 1 Processor -- start 2024-03-25T19:53:11.094 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.094+0000 7f5106833640 1 -- start start 2024-03-25T19:53:11.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.094+0000 7f5106833640 10 monclient: init 2024-03-25T19:53:11.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.094+0000 7f5106833640 10 monclient: _reopen_session rank -1 2024-03-25T19:53:11.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.094+0000 7f5106833640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:53:11.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.094+0000 7f5106833640 10 monclient(hunting): picked mon.noname-c con 0x7f51000a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:53:11.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.094+0000 7f5106833640 10 monclient(hunting): picked mon.noname-a con 0x7f510013f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:53:11.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.094+0000 7f5106833640 10 monclient(hunting): picked mon.noname-b con 0x7f51001430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:53:11.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.095+0000 7f5106833640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5100142520 con 0x7f510013f710 2024-03-25T19:53:11.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.095+0000 7f5106833640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51000b28a0 con 0x7f51000a98f0 2024-03-25T19:53:11.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.095+0000 7f5106833640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51000b2a30 con 0x7f51001430c0 2024-03-25T19:53:11.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.095+0000 7f5106833640 10 monclient(hunting): _renew_subs 2024-03-25T19:53:11.095 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.095+0000 7f5106833640 10 monclient(hunting): authenticate will time out at 4626.177043s 2024-03-25T19:53:11.104 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.105+0000 7f5104dd0640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f51001430c0 0x7f510014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:58714/0 (socket says 172.21.15.81:58714) 2024-03-25T19:53:11.105 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.105+0000 7f5104dd0640 1 -- 172.21.15.81:0/4285761725 learned_addr learned my addr 172.21.15.81:0/4285761725 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:53:11.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.112+0000 7f50feffd640 1 -- 172.21.15.81:0/4285761725 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1175422962 0 0) 0x7f51000b28a0 con 0x7f51000a98f0 2024-03-25T19:53:11.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.112+0000 7f50feffd640 10 monclient(hunting): _negotiate 2024-03-25T19:53:11.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.112+0000 7f50feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:11.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.112+0000 7f50feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:11.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.112+0000 7f50feffd640 10 monclient(hunting): my global_id is 4717 2024-03-25T19:53:11.112 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.112+0000 7f50feffd640 1 -- 172.21.15.81:0/4285761725 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50e8003520 con 0x7f51000a98f0 2024-03-25T19:53:11.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f50feffd640 1 -- 172.21.15.81:0/4285761725 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 107198975 0 0) 0x7f50e8003520 con 0x7f51000a98f0 2024-03-25T19:53:11.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f50feffd640 1 -- 172.21.15.81:0/4285761725 >> v1:172.21.15.81:6790/0 conn(0x7f51001430c0 legacy=0x7f510014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:11.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f50feffd640 1 -- 172.21.15.81:0/4285761725 >> v1:172.21.15.81:6789/0 conn(0x7f510013f710 legacy=0x7f51001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:11.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f50feffd640 10 monclient: _finish_hunting 0 2024-03-25T19:53:11.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f50feffd640 1 monclient: found mon.noname-c 2024-03-25T19:53:11.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f50feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:53:11.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f50feffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:53:11.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f50feffd640 1 -- 172.21.15.81:0/4285761725 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51000b44c0 con 0x7f51000a98f0 2024-03-25T19:53:11.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f50feffd640 10 monclient: _finish_auth 0 2024-03-25T19:53:11.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f50feffd640 10 monclient: _check_auth_tickets 2024-03-25T19:53:11.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f50feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:53:11.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f50feffd640 1 -- 172.21.15.81:0/4285761725 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f50f4001690 con 0x7f51000a98f0 2024-03-25T19:53:11.119 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f5106833640 5 monclient: authenticate success, global_id 4717 2024-03-25T19:53:11.120 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f5106833640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:53:11.120 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f50feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:11.120 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f50feffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:53:11.120 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f50feffd640 10 monclient: dump: 2024-03-25T19:53:11.120 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:53:11.120 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:11.120 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:11.120 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:11.120 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:11.120 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:11.120 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:53:11.120 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:53:11.120 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:53:11.121 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:11.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.119+0000 7f5106833640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:53:11.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.134+0000 7f50feffd640 1 -- 172.21.15.81:0/4285761725 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50f40017f0 con 0x7f51000a98f0 2024-03-25T19:53:11.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.134+0000 7f50feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:53:11.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.134+0000 7f50feffd640 1 -- 172.21.15.81:0/4285761725 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f50f4001c00 con 0x7f51000a98f0 2024-03-25T19:53:11.134 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.134+0000 7f5106833640 10 monclient: get_monmap_and_config success 2024-03-25T19:53:11.135 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.134+0000 7f5106833640 4 set_mon_vals no callback set 2024-03-25T19:53:11.135 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.134+0000 7f50feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:11.135 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.134+0000 7f50feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:53:11.135 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.134+0000 7f50feffd640 10 monclient: dump: 2024-03-25T19:53:11.135 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:53:11.135 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:11.135 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:11.135 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:11.135 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:11.135 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:11.135 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:53:11.135 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:53:11.135 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:53:11.135 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:11.138 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.135+0000 7f5106833640 10 monclient: shutdown 2024-03-25T19:53:11.138 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.135+0000 7f5106833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:53:11.138 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.135+0000 7f5106833640 1 -- 172.21.15.81:0/4285761725 >> v1:172.21.15.98:6789/0 conn(0x7f51000a98f0 legacy=0x7f51000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:11.138 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.135+0000 7f5106833640 1 -- 172.21.15.81:0/4285761725 shutdown_connections 2024-03-25T19:53:11.138 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.135+0000 7f5106833640 1 -- 172.21.15.81:0/4285761725 >> 172.21.15.81:0/4285761725 conn(0x7f51000b7ea0 msgr2=0x7f51000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:53:11.138 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.135+0000 7f5106833640 1 -- 172.21.15.81:0/4285761725 shutdown_connections 2024-03-25T19:53:11.138 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.135+0000 7f5106833640 1 -- 172.21.15.81:0/4285761725 wait complete. 2024-03-25T19:53:11.138 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.136+0000 7f5106833640 10 monclient: build_initial_monmap 2024-03-25T19:53:11.138 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.136+0000 7f5106833640 10 monclient: monmap: 2024-03-25T19:53:11.138 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:53:11.138 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:53:11.138 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:53:11.139 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:53:11.139 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:53:11.139 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:53:11.139 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:53:11.139 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:53:11.139 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:53:11.139 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:11.139 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.136+0000 7f5106833640 1 librados: starting msgr at 2024-03-25T19:53:11.139 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.136+0000 7f5106833640 1 librados: starting objecter 2024-03-25T19:53:11.139 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.137+0000 7f5106833640 1 Processor -- start 2024-03-25T19:53:11.139 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.137+0000 7f5106833640 1 -- start start 2024-03-25T19:53:11.139 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.137+0000 7f5106833640 1 librados: setting wanted keys 2024-03-25T19:53:11.139 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.137+0000 7f5106833640 1 librados: calling monclient init 2024-03-25T19:53:11.139 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.137+0000 7f5106833640 10 monclient: init 2024-03-25T19:53:11.140 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.138+0000 7f5106833640 10 monclient: _reopen_session rank -1 2024-03-25T19:53:11.140 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.138+0000 7f5106833640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:53:11.140 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.138+0000 7f5106833640 10 monclient(hunting): picked mon.noname-a con 0x7f51000a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:53:11.140 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.138+0000 7f5106833640 10 monclient(hunting): picked mon.noname-b con 0x7f510013f710 addr v1:172.21.15.98:6789/0 2024-03-25T19:53:11.140 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.138+0000 7f5106833640 10 monclient(hunting): picked mon.noname-c con 0x7f51001430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:53:11.140 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.138+0000 7f5106833640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51000ab9e0 con 0x7f51000a98f0 2024-03-25T19:53:11.140 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.138+0000 7f5106833640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5100138c20 con 0x7f510013f710 2024-03-25T19:53:11.140 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.138+0000 7f5106833640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5100138f00 con 0x7f51001430c0 2024-03-25T19:53:11.140 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.138+0000 7f5106833640 10 monclient(hunting): _renew_subs 2024-03-25T19:53:11.140 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.138+0000 7f5106833640 10 monclient(hunting): authenticate will time out at 4626.220094s 2024-03-25T19:53:11.147 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.148+0000 7f5104dd0640 1 --1- >> v1:172.21.15.81:6790/0 conn(0x7f51001430c0 0x7f51001e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6790/0 says I am v1:172.21.15.81:58754/0 (socket says 172.21.15.81:58754) 2024-03-25T19:53:11.147 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.148+0000 7f5104dd0640 1 -- 172.21.15.81:0/371353614 learned_addr learned my addr 172.21.15.81:0/371353614 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:53:11.149 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.149+0000 7f50fcff9640 10 client.?.objecter ms_handle_connect 0x7f51001430c0 2024-03-25T19:53:11.149 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.149+0000 7f50fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:11.151 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.152+0000 7f50fcff9640 10 client.?.objecter ms_handle_connect 0x7f51000a98f0 2024-03-25T19:53:11.151 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.152+0000 7f50fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:11.155 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.156+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656031588 0 0) 0x7f5100138f00 con 0x7f51001430c0 2024-03-25T19:53:11.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.156+0000 7f50fcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:53:11.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.156+0000 7f50fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:11.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.156+0000 7f50fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:11.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.156+0000 7f50fcff9640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:53:11.156 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.156+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50ec004eb0 con 0x7f51001430c0 2024-03-25T19:53:11.157 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.157+0000 7f50fcff9640 10 client.?.objecter ms_handle_connect 0x7f510013f710 2024-03-25T19:53:11.157 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.158+0000 7f50fcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:11.164 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.165+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 <== mon.2 v1:172.21.15.81:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 238220255 0 0) 0x7f50ec004eb0 con 0x7f51001430c0 2024-03-25T19:53:11.164 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.165+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 --> v1:172.21.15.81:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5100138f00 con 0x7f51001430c0 2024-03-25T19:53:11.165 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.165+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 <== mon.2 v1:172.21.15.81:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f50f8001b10 con 0x7f51001430c0 2024-03-25T19:53:11.165 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.165+0000 7f50fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:11.165 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.165+0000 7f50fcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:53:11.165 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.165+0000 7f50fcff9640 10 monclient(hunting): dump: 2024-03-25T19:53:11.165 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:53:11.165 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:11.165 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:11.165 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:11.165 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:11.165 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:11.165 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:53:11.165 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:53:11.165 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:53:11.165 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:11.165 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.165+0000 7f50fcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:53:11.166 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.166+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034995404 0 0) 0x7f5100138c20 con 0x7f510013f710 2024-03-25T19:53:11.166 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.166+0000 7f50fcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:53:11.166 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.166+0000 7f50fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:11.166 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.166+0000 7f50fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:11.166 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.166+0000 7f50fcff9640 10 monclient(hunting): my global_id is 4720 2024-03-25T19:53:11.166 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.166+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50ec004eb0 con 0x7f510013f710 2024-03-25T19:53:11.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.174+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831271109 0 0) 0x7f51000ab9e0 con 0x7f51000a98f0 2024-03-25T19:53:11.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.174+0000 7f50fcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:53:11.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.174+0000 7f50fcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:11.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.174+0000 7f50fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:11.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.174+0000 7f50fcff9640 10 monclient(hunting): my global_id is 4746 2024-03-25T19:53:11.174 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.174+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5100138c20 con 0x7f51000a98f0 2024-03-25T19:53:11.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 <== mon.2 v1:172.21.15.81:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2986394417 0 0) 0x7f5100138f00 con 0x7f51001430c0 2024-03-25T19:53:11.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 >> v1:172.21.15.98:6789/0 conn(0x7f510013f710 legacy=0x7f51001d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:11.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 >> v1:172.21.15.81:6789/0 conn(0x7f51000a98f0 legacy=0x7f5100137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:11.180 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f50fcff9640 10 monclient: _finish_hunting 0 2024-03-25T19:53:11.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f50fcff9640 1 monclient: found mon.c 2024-03-25T19:53:11.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f50fcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:53:11.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f50fcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:53:11.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 --> v1:172.21.15.81:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510014e7a0 con 0x7f51001430c0 2024-03-25T19:53:11.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f50fcff9640 10 monclient: _finish_auth 0 2024-03-25T19:53:11.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f50fcff9640 10 monclient: _check_auth_tickets 2024-03-25T19:53:11.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f50fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:53:11.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f5106833640 5 monclient: authenticate success, global_id 4703 2024-03-25T19:53:11.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f5106833640 10 monclient: _renew_subs 2024-03-25T19:53:11.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f5106833640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:53:11.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f5106833640 1 -- 172.21.15.81:0/371353614 --> v1:172.21.15.81:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51001386d0 con 0x7f51001430c0 2024-03-25T19:53:11.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f5106833640 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:53:11.181 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f5106833640 10 monclient: _renew_subs 2024-03-25T19:53:11.182 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f5106833640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:53:11.182 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f5106833640 1 -- 172.21.15.81:0/371353614 --> v1:172.21.15.81:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51001d8060 con 0x7f51001430c0 2024-03-25T19:53:11.182 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.180+0000 7f5106833640 1 librados: init done 2024-03-25T19:53:11.182 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.181+0000 7f5106833640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:11.182 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.181+0000 7f5106833640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:11.183 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.181+0000 7f5106833640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:53:11.183 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.181+0000 7f5106833640 1 -- 172.21.15.81:0/371353614 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51000627b0 con 0x7f51001430c0 2024-03-25T19:53:11.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.511+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 <== mon.2 v1:172.21.15.81:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50f8002040 con 0x7f51001430c0 2024-03-25T19:53:11.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.511+0000 7f50fcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:53:11.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.511+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 <== mon.2 v1:172.21.15.81:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f50f8004d00 con 0x7f51001430c0 2024-03-25T19:53:11.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.511+0000 7f50fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:11.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.511+0000 7f50fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:53:11.511 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.511+0000 7f50fcff9640 10 monclient: dump: 2024-03-25T19:53:11.511 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:53:11.511 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:11.511 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:11.511 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:11.511 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:11.512 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:11.512 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:53:11.512 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:53:11.512 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:53:11.512 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:11.512 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.511+0000 7f50fe7fc640 4 set_mon_vals no callback set 2024-03-25T19:53:11.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.512+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 <== mon.2 v1:172.21.15.81:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f50f8004ee0 con 0x7f51001430c0 2024-03-25T19:53:11.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.512+0000 7f50fffff640 10 monclient: get_auth_request con 0x7f50ec06de60 auth_method 0 2024-03-25T19:53:11.513 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.513+0000 7f50fcff9640 10 client.4703.objecter ms_handle_connect 0x7f50ec06de60 2024-03-25T19:53:11.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.517+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 <== mon.2 v1:172.21.15.81:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3447814714 0 0) 0x7f50f8054a80 con 0x7f51001430c0 2024-03-25T19:53:11.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.517+0000 7f50fcff9640 10 client.4703.objecter ms_dispatch 0x7f5100000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:53:11.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.517+0000 7f50fcff9640 3 client.4703.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:53:11.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.517+0000 7f50fcff9640 3 client.4703.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:53:11.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.517+0000 7f50fcff9640 20 client.4703.objecter dump_active .. 0 homeless 2024-03-25T19:53:11.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.517+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 <== mon.2 v1:172.21.15.81:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f50f80503c0 con 0x7f51001430c0 2024-03-25T19:53:11.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.517+0000 7f50fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:11.517 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.517+0000 7f50fcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:53:11.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.777+0000 7f5106833640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:11.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.777+0000 7f5106833640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:11.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.777+0000 7f5106833640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.81:6790/0 2024-03-25T19:53:11.777 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.777+0000 7f5106833640 1 -- 172.21.15.81:0/371353614 --> v1:172.21.15.81:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f510014f240 con 0x7f51001430c0 2024-03-25T19:53:11.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.805+0000 7f50fcff9640 1 -- 172.21.15.81:0/371353614 <== mon.2 v1:172.21.15.81:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3235219182) 0x7f50f80524c0 con 0x7f51001430c0 2024-03-25T19:53:11.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.805+0000 7f50fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:11.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.805+0000 7f50fcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:53:11.805 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:53:11.805 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":11,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:52:03.690842+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:52:03.690842+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:53:11.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.807+0000 7f5106833640 10 librados: watch_flush enter 2024-03-25T19:53:11.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.808+0000 7f5106833640 10 librados: watch_flush exit 2024-03-25T19:53:11.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.808+0000 7f5106833640 20 client.4703.objecter shutdown clearing up homeless session... 2024-03-25T19:53:11.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.808+0000 7f5106833640 10 client.4703.objecter successfully canceled tick 2024-03-25T19:53:11.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.808+0000 7f5106833640 1 -- 172.21.15.81:0/371353614 >> v1:172.21.15.98:6800/3980829775 conn(0x7f50ec06de60 legacy=0x7f50ec070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:11.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.808+0000 7f5106833640 10 monclient: shutdown 2024-03-25T19:53:11.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.808+0000 7f5106833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:53:11.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.808+0000 7f5106833640 1 -- 172.21.15.81:0/371353614 >> v1:172.21.15.81:6790/0 conn(0x7f51001430c0 legacy=0x7f51001e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:11.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.808+0000 7f5106833640 1 -- 172.21.15.81:0/371353614 shutdown_connections 2024-03-25T19:53:11.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.808+0000 7f5106833640 1 -- 172.21.15.81:0/371353614 >> 172.21.15.81:0/371353614 conn(0x7f51000b7ea0 msgr2=0x7f5100074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:53:11.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.808+0000 7f5106833640 1 -- 172.21.15.81:0/371353614 shutdown_connections 2024-03-25T19:53:11.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.808+0000 7f5106833640 1 -- 172.21.15.81:0/371353614 wait complete. 2024-03-25T19:53:11.809 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:11.808+0000 7f5106833640 1 librados: shutdown 2024-03-25T19:53:11.819 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:53:18.625 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:53:18.744 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.744+0000 7fee79f9d640 10 monclient: get_monmap_and_config 2024-03-25T19:53:18.744 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.744+0000 7fee79f9d640 10 monclient: build_initial_monmap 2024-03-25T19:53:18.744 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.744+0000 7fee79f9d640 10 monclient: monmap: 2024-03-25T19:53:18.744 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:53:18.744 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:18.744 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:53:18.745722+0000 2024-03-25T19:53:18.745 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:53:18.745722+0000 2024-03-25T19:53:18.745 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:53:18.745 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:18.745 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:53:18.745 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:53:18.745 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:53:18.745 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:18.746 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.745+0000 7fee79f9d640 1 Processor -- start 2024-03-25T19:53:18.746 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.745+0000 7fee79f9d640 1 -- start start 2024-03-25T19:53:18.746 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.745+0000 7fee79f9d640 10 monclient: init 2024-03-25T19:53:18.746 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.746+0000 7fee79f9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:53:18.746 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.746+0000 7fee79f9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:53:18.747 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.746+0000 7fee79f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fee740a98f0 addr v1:172.21.15.81:6789/0 2024-03-25T19:53:18.747 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.746+0000 7fee79f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fee7413f710 addr v1:172.21.15.81:6790/0 2024-03-25T19:53:18.747 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.746+0000 7fee79f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fee741430c0 addr v1:172.21.15.98:6789/0 2024-03-25T19:53:18.747 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.746+0000 7fee79f9d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee74142520 con 0x7fee740a98f0 2024-03-25T19:53:18.747 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.746+0000 7fee79f9d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee740b28a0 con 0x7fee741430c0 2024-03-25T19:53:18.747 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.746+0000 7fee79f9d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee740b2a30 con 0x7fee7413f710 2024-03-25T19:53:18.747 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.746+0000 7fee79f9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:53:18.747 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.746+0000 7fee79f9d640 10 monclient(hunting): authenticate will time out at 4633.828213s 2024-03-25T19:53:18.755 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.756+0000 7fee73fff640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7fee741430c0 0x7fee7414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:34324/0 (socket says 172.21.15.81:34324) 2024-03-25T19:53:18.756 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.756+0000 7fee73fff640 1 -- 172.21.15.81:0/861570234 learned_addr learned my addr 172.21.15.81:0/861570234 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:53:18.767 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.767+0000 7fee727fc640 1 -- 172.21.15.81:0/861570234 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957498700 0 0) 0x7fee740b28a0 con 0x7fee741430c0 2024-03-25T19:53:18.767 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.767+0000 7fee727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:53:18.767 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.767+0000 7fee727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:18.767 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.767+0000 7fee727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:18.767 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.767+0000 7fee727fc640 10 monclient(hunting): my global_id is 4723 2024-03-25T19:53:18.768 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.767+0000 7fee727fc640 1 -- 172.21.15.81:0/861570234 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee5c003520 con 0x7fee741430c0 2024-03-25T19:53:18.769 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.769+0000 7fee727fc640 1 -- 172.21.15.81:0/861570234 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330636911 0 0) 0x7fee74142520 con 0x7fee740a98f0 2024-03-25T19:53:18.769 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.769+0000 7fee727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:53:18.769 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.769+0000 7fee727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:18.769 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.769+0000 7fee727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:18.769 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.769+0000 7fee727fc640 10 monclient(hunting): my global_id is 4749 2024-03-25T19:53:18.769 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.769+0000 7fee727fc640 1 -- 172.21.15.81:0/861570234 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee740b28a0 con 0x7fee740a98f0 2024-03-25T19:53:18.771 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.771+0000 7fee727fc640 1 -- 172.21.15.81:0/861570234 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305118987 0 0) 0x7fee740b2a30 con 0x7fee7413f710 2024-03-25T19:53:18.771 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.771+0000 7fee727fc640 10 monclient(hunting): _negotiate 2024-03-25T19:53:18.771 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.771+0000 7fee727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:18.771 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.771+0000 7fee727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:18.771 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.771+0000 7fee727fc640 10 monclient(hunting): my global_id is 4706 2024-03-25T19:53:18.771 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.771+0000 7fee727fc640 1 -- 172.21.15.81:0/861570234 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee74142520 con 0x7fee7413f710 2024-03-25T19:53:18.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee727fc640 1 -- 172.21.15.81:0/861570234 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3354054143 0 0) 0x7fee740b28a0 con 0x7fee740a98f0 2024-03-25T19:53:18.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee727fc640 1 -- 172.21.15.81:0/861570234 >> v1:172.21.15.81:6790/0 conn(0x7fee7413f710 legacy=0x7fee741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:18.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee727fc640 1 -- 172.21.15.81:0/861570234 >> v1:172.21.15.98:6789/0 conn(0x7fee741430c0 legacy=0x7fee7414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:18.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee727fc640 10 monclient: _finish_hunting 0 2024-03-25T19:53:18.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee727fc640 1 monclient: found mon.noname-a 2024-03-25T19:53:18.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:53:18.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee727fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.81:6789/0 2024-03-25T19:53:18.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee727fc640 1 -- 172.21.15.81:0/861570234 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee740b44c0 con 0x7fee740a98f0 2024-03-25T19:53:18.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee727fc640 10 monclient: _finish_auth 0 2024-03-25T19:53:18.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee727fc640 10 monclient: _check_auth_tickets 2024-03-25T19:53:18.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:53:18.782 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee727fc640 1 -- 172.21.15.81:0/861570234 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fee64001670 con 0x7fee740a98f0 2024-03-25T19:53:18.783 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee79f9d640 5 monclient: authenticate success, global_id 4749 2024-03-25T19:53:18.783 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee79f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:53:18.783 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:18.783 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:53:18.783 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee727fc640 10 monclient: dump: 2024-03-25T19:53:18.783 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:53:18.783 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:18.783 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:18.783 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:18.783 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:18.783 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:18.783 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:53:18.783 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:53:18.783 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:53:18.783 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:18.784 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.782+0000 7fee79f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:53:18.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.802+0000 7fee727fc640 1 -- 172.21.15.81:0/861570234 <== mon.0 v1:172.21.15.81:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee640017d0 con 0x7fee740a98f0 2024-03-25T19:53:18.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.802+0000 7fee727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:53:18.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.802+0000 7fee727fc640 1 -- 172.21.15.81:0/861570234 <== mon.0 v1:172.21.15.81:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fee64001be0 con 0x7fee740a98f0 2024-03-25T19:53:18.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.802+0000 7fee79f9d640 10 monclient: get_monmap_and_config success 2024-03-25T19:53:18.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.802+0000 7fee79f9d640 4 set_mon_vals no callback set 2024-03-25T19:53:18.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.802+0000 7fee727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:18.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.802+0000 7fee727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:53:18.802 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.802+0000 7fee727fc640 10 monclient: dump: 2024-03-25T19:53:18.802 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:53:18.802 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:18.802 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:18.803 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:18.803 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:18.803 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:18.803 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:53:18.803 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:53:18.803 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:53:18.803 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:18.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.803+0000 7fee79f9d640 10 monclient: shutdown 2024-03-25T19:53:18.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.803+0000 7fee79f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:53:18.805 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.803+0000 7fee79f9d640 1 -- 172.21.15.81:0/861570234 >> v1:172.21.15.81:6789/0 conn(0x7fee740a98f0 legacy=0x7fee740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:18.806 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.803+0000 7fee79f9d640 1 -- 172.21.15.81:0/861570234 shutdown_connections 2024-03-25T19:53:18.806 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.803+0000 7fee79f9d640 1 -- 172.21.15.81:0/861570234 >> 172.21.15.81:0/861570234 conn(0x7fee740b7ea0 msgr2=0x7fee740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:53:18.806 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.803+0000 7fee79f9d640 1 -- 172.21.15.81:0/861570234 shutdown_connections 2024-03-25T19:53:18.806 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.803+0000 7fee79f9d640 1 -- 172.21.15.81:0/861570234 wait complete. 2024-03-25T19:53:18.806 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.804+0000 7fee79f9d640 10 monclient: build_initial_monmap 2024-03-25T19:53:18.806 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.804+0000 7fee79f9d640 10 monclient: monmap: 2024-03-25T19:53:18.806 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:53:18.806 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:53:18.806 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:53:18.806 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:53:18.806 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:53:18.806 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:53:18.806 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:53:18.806 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:53:18.807 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:53:18.807 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:18.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.804+0000 7fee79f9d640 1 librados: starting msgr at 2024-03-25T19:53:18.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.804+0000 7fee79f9d640 1 librados: starting objecter 2024-03-25T19:53:18.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.804+0000 7fee79f9d640 1 Processor -- start 2024-03-25T19:53:18.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.805+0000 7fee79f9d640 1 -- start start 2024-03-25T19:53:18.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.805+0000 7fee79f9d640 1 librados: setting wanted keys 2024-03-25T19:53:18.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.805+0000 7fee79f9d640 1 librados: calling monclient init 2024-03-25T19:53:18.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.805+0000 7fee79f9d640 10 monclient: init 2024-03-25T19:53:18.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.805+0000 7fee79f9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:53:18.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.805+0000 7fee79f9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:53:18.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.805+0000 7fee79f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fee740a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:53:18.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.805+0000 7fee79f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fee7413f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:53:18.807 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.805+0000 7fee79f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fee741430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:53:18.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.805+0000 7fee79f9d640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee74137030 con 0x7fee7413f710 2024-03-25T19:53:18.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.805+0000 7fee79f9d640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee741398a0 con 0x7fee740a98f0 2024-03-25T19:53:18.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.805+0000 7fee79f9d640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee741d4c90 con 0x7fee741430c0 2024-03-25T19:53:18.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.805+0000 7fee79f9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:53:18.808 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.805+0000 7fee79f9d640 10 monclient(hunting): authenticate will time out at 4633.887746s 2024-03-25T19:53:18.815 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.815+0000 7fee72ffd640 1 --1- >> v1:172.21.15.81:6789/0 conn(0x7fee7413f710 0x7fee740aff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.81:6789/0 says I am v1:172.21.15.81:32894/0 (socket says 172.21.15.81:32894) 2024-03-25T19:53:18.815 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.815+0000 7fee72ffd640 1 -- 172.21.15.81:0/3745231943 learned_addr learned my addr 172.21.15.81:0/3745231943 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:53:18.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.818+0000 7fee5bfff640 10 client.?.objecter ms_handle_connect 0x7fee741430c0 2024-03-25T19:53:18.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.818+0000 7fee5bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:18.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.818+0000 7fee5bfff640 10 client.?.objecter ms_handle_connect 0x7fee7413f710 2024-03-25T19:53:18.818 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.818+0000 7fee5bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:18.820 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.821+0000 7fee5bfff640 10 client.?.objecter ms_handle_connect 0x7fee740a98f0 2024-03-25T19:53:18.821 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.821+0000 7fee5bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:18.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.828+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201713713 0 0) 0x7fee741398a0 con 0x7fee740a98f0 2024-03-25T19:53:18.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.828+0000 7fee5bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:53:18.827 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.828+0000 7fee5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:18.828 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.828+0000 7fee5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:18.828 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.828+0000 7fee5bfff640 10 monclient(hunting): my global_id is 4726 2024-03-25T19:53:18.828 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.828+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee60004eb0 con 0x7fee740a98f0 2024-03-25T19:53:18.829 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.830+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679648680 0 0) 0x7fee74137030 con 0x7fee7413f710 2024-03-25T19:53:18.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.830+0000 7fee5bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:53:18.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.830+0000 7fee5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:18.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.830+0000 7fee5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:18.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.830+0000 7fee5bfff640 10 monclient(hunting): my global_id is 4752 2024-03-25T19:53:18.830 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.830+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee741398a0 con 0x7fee7413f710 2024-03-25T19:53:18.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.832+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2718799271 0 0) 0x7fee741d4c90 con 0x7fee741430c0 2024-03-25T19:53:18.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.832+0000 7fee5bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:53:18.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.832+0000 7fee5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:18.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.832+0000 7fee5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:18.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.832+0000 7fee5bfff640 10 monclient(hunting): my global_id is 4709 2024-03-25T19:53:18.832 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.832+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee74137030 con 0x7fee741430c0 2024-03-25T19:53:18.837 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.838+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2794736065 0 0) 0x7fee60004eb0 con 0x7fee740a98f0 2024-03-25T19:53:18.837 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.838+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 --> v1:172.21.15.98:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee741d4c90 con 0x7fee740a98f0 2024-03-25T19:53:18.837 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.838+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fee64001ba0 con 0x7fee740a98f0 2024-03-25T19:53:18.838 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.838+0000 7fee5bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:18.838 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.838+0000 7fee5bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:53:18.838 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.838+0000 7fee5bfff640 10 monclient(hunting): dump: 2024-03-25T19:53:18.838 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:53:18.838 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:18.838 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:18.838 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:18.838 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:18.838 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:18.838 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:53:18.838 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:53:18.838 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:53:18.838 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:18.838 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.838+0000 7fee5bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:53:18.843 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 <== mon.1 v1:172.21.15.98:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2109820670 0 0) 0x7fee741d4c90 con 0x7fee740a98f0 2024-03-25T19:53:18.843 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 >> v1:172.21.15.81:6790/0 conn(0x7fee741430c0 legacy=0x7fee741d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:18.843 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 >> v1:172.21.15.81:6789/0 conn(0x7fee7413f710 legacy=0x7fee740aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:18.843 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee5bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:53:18.843 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee5bfff640 1 monclient: found mon.b 2024-03-25T19:53:18.843 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:53:18.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee5bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:53:18.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee740b16b0 con 0x7fee740a98f0 2024-03-25T19:53:18.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee5bfff640 10 monclient: _finish_auth 0 2024-03-25T19:53:18.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee5bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:53:18.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:53:18.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee79f9d640 5 monclient: authenticate success, global_id 4726 2024-03-25T19:53:18.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee79f9d640 10 monclient: _renew_subs 2024-03-25T19:53:18.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee79f9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:53:18.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee79f9d640 1 -- 172.21.15.81:0/3745231943 --> v1:172.21.15.98:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee740b0c10 con 0x7fee740a98f0 2024-03-25T19:53:18.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee79f9d640 10 client.4726.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:53:18.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee79f9d640 10 monclient: _renew_subs 2024-03-25T19:53:18.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee79f9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:53:18.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee79f9d640 1 -- 172.21.15.81:0/3745231943 --> v1:172.21.15.98:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee740b0610 con 0x7fee740a98f0 2024-03-25T19:53:18.844 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.843+0000 7fee79f9d640 1 librados: init done 2024-03-25T19:53:18.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.844+0000 7fee79f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:18.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.844+0000 7fee79f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:18.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.844+0000 7fee79f9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:53:18.845 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:18.844+0000 7fee79f9d640 1 -- 172.21.15.81:0/3745231943 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee741d4e20 con 0x7fee740a98f0 2024-03-25T19:53:19.116 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.116+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 <== mon.1 v1:172.21.15.98:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee640038a0 con 0x7fee740a98f0 2024-03-25T19:53:19.116 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.116+0000 7fee5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:53:19.116 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.116+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 <== mon.1 v1:172.21.15.98:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7fee64003bb0 con 0x7fee740a98f0 2024-03-25T19:53:19.116 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.116+0000 7fee5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:19.116 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.116+0000 7fee5bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:53:19.116 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.116+0000 7fee5bfff640 10 monclient: dump: 2024-03-25T19:53:19.116 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:53:19.116 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:19.116 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:19.116 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:19.116 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:19.116 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:19.116 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:53:19.117 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:53:19.117 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:53:19.117 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:19.117 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.116+0000 7fee71ffb640 4 set_mon_vals no callback set 2024-03-25T19:53:19.117 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.116+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 <== mon.1 v1:172.21.15.98:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7fee64019b80 con 0x7fee740a98f0 2024-03-25T19:53:19.117 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.117+0000 7fee72ffd640 10 monclient: get_auth_request con 0x7fee6006de50 auth_method 0 2024-03-25T19:53:19.117 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.117+0000 7fee5bfff640 10 client.4726.objecter ms_handle_connect 0x7fee6006de50 2024-03-25T19:53:19.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.121+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 <== mon.1 v1:172.21.15.98:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3447814714 0 0) 0x7fee64053700 con 0x7fee740a98f0 2024-03-25T19:53:19.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.121+0000 7fee5bfff640 10 client.4726.objecter ms_dispatch 0x7fee74000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:53:19.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.121+0000 7fee5bfff640 3 client.4726.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:53:19.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.121+0000 7fee5bfff640 3 client.4726.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:53:19.121 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.122+0000 7fee5bfff640 20 client.4726.objecter dump_active .. 0 homeless 2024-03-25T19:53:19.122 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.122+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 <== mon.1 v1:172.21.15.98:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee6404f040 con 0x7fee740a98f0 2024-03-25T19:53:19.122 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.122+0000 7fee5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:19.122 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.122+0000 7fee5bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:53:19.367 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.368+0000 7fee79f9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:19.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.368+0000 7fee79f9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:19.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.368+0000 7fee79f9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.98:6789/0 2024-03-25T19:53:19.368 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.368+0000 7fee79f9d640 1 -- 172.21.15.81:0/3745231943 --> v1:172.21.15.98:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee740627b0 con 0x7fee740a98f0 2024-03-25T19:53:19.399 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.400+0000 7fee5bfff640 1 -- 172.21.15.81:0/3745231943 <== mon.1 v1:172.21.15.98:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3235219182) 0x7fee64051140 con 0x7fee740a98f0 2024-03-25T19:53:19.399 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.400+0000 7fee5bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:19.400 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.400+0000 7fee5bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:53:19.400 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:53:19.400 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":11,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:52:03.690842+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:52:03.690842+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:53:19.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.402+0000 7fee79f9d640 10 librados: watch_flush enter 2024-03-25T19:53:19.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.402+0000 7fee79f9d640 10 librados: watch_flush exit 2024-03-25T19:53:19.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.402+0000 7fee79f9d640 20 client.4726.objecter shutdown clearing up homeless session... 2024-03-25T19:53:19.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.402+0000 7fee79f9d640 10 client.4726.objecter successfully canceled tick 2024-03-25T19:53:19.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.402+0000 7fee79f9d640 1 -- 172.21.15.81:0/3745231943 >> v1:172.21.15.98:6800/3980829775 conn(0x7fee6006de50 legacy=0x7fee60070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:19.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.402+0000 7fee79f9d640 10 monclient: shutdown 2024-03-25T19:53:19.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.402+0000 7fee79f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:53:19.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.402+0000 7fee79f9d640 1 -- 172.21.15.81:0/3745231943 >> v1:172.21.15.98:6789/0 conn(0x7fee740a98f0 legacy=0x7fee741d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:19.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.402+0000 7fee79f9d640 1 -- 172.21.15.81:0/3745231943 shutdown_connections 2024-03-25T19:53:19.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.402+0000 7fee79f9d640 1 -- 172.21.15.81:0/3745231943 >> 172.21.15.81:0/3745231943 conn(0x7fee740b7ea0 msgr2=0x7fee74074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:53:19.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.403+0000 7fee79f9d640 1 -- 172.21.15.81:0/3745231943 shutdown_connections 2024-03-25T19:53:19.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.403+0000 7fee79f9d640 1 -- 172.21.15.81:0/3745231943 wait complete. 2024-03-25T19:53:19.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:19.403+0000 7fee79f9d640 1 librados: shutdown 2024-03-25T19:53:19.413 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:53:26.219 DEBUG:teuthology.orchestra.run.smithi081:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:53:26.335 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.336+0000 7f7781663640 10 monclient: get_monmap_and_config 2024-03-25T19:53:26.335 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.336+0000 7f7781663640 10 monclient: build_initial_monmap 2024-03-25T19:53:26.335 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.336+0000 7f7781663640 10 monclient: monmap: 2024-03-25T19:53:26.336 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:53:26.336 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:26.336 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:53:26.336983+0000 2024-03-25T19:53:26.336 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:53:26.336983+0000 2024-03-25T19:53:26.336 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:53:26.336 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:26.336 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:53:26.336 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-c 2024-03-25T19:53:26.336 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-b 2024-03-25T19:53:26.336 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:26.338 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.336+0000 7f7781663640 1 Processor -- start 2024-03-25T19:53:26.338 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.336+0000 7f7781663640 1 -- start start 2024-03-25T19:53:26.338 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.336+0000 7f7781663640 10 monclient: init 2024-03-25T19:53:26.338 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.337+0000 7f7781663640 10 monclient: _reopen_session rank -1 2024-03-25T19:53:26.338 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.337+0000 7f7781663640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:53:26.338 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.337+0000 7f7781663640 10 monclient(hunting): picked mon.noname-c con 0x7f777c0a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:53:26.338 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.337+0000 7f7781663640 10 monclient(hunting): picked mon.noname-a con 0x7f777c13f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:53:26.338 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.337+0000 7f7781663640 10 monclient(hunting): picked mon.noname-b con 0x7f777c1430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:53:26.338 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.337+0000 7f7781663640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f777c142520 con 0x7f777c13f710 2024-03-25T19:53:26.338 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.337+0000 7f7781663640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f777c0b28a0 con 0x7f777c0a98f0 2024-03-25T19:53:26.338 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.337+0000 7f7781663640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f777c0b2a30 con 0x7f777c1430c0 2024-03-25T19:53:26.339 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.337+0000 7f7781663640 10 monclient(hunting): _renew_subs 2024-03-25T19:53:26.339 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.337+0000 7f7781663640 10 monclient(hunting): authenticate will time out at 4641.419547s 2024-03-25T19:53:26.345 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.346+0000 7f777affd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f777c0a98f0 0x7f777c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:39076/0 (socket says 172.21.15.81:39076) 2024-03-25T19:53:26.345 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.346+0000 7f777affd640 1 -- 172.21.15.81:0/4048446953 learned_addr learned my addr 172.21.15.81:0/4048446953 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:53:26.354 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.355+0000 7f7779ffb640 1 -- 172.21.15.81:0/4048446953 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576532837 0 0) 0x7f777c0b28a0 con 0x7f777c0a98f0 2024-03-25T19:53:26.355 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.355+0000 7f7779ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:53:26.355 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.355+0000 7f7779ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:26.355 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.355+0000 7f7779ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:26.355 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.355+0000 7f7779ffb640 10 monclient(hunting): my global_id is 4729 2024-03-25T19:53:26.355 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.355+0000 7f7779ffb640 1 -- 172.21.15.81:0/4048446953 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7764003520 con 0x7f777c0a98f0 2024-03-25T19:53:26.379 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.379+0000 7f7779ffb640 1 -- 172.21.15.81:0/4048446953 <== mon.1 v1:172.21.15.98:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4038479970 0 0) 0x7f7764003520 con 0x7f777c0a98f0 2024-03-25T19:53:26.379 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.379+0000 7f7779ffb640 1 -- 172.21.15.81:0/4048446953 >> v1:172.21.15.81:6790/0 conn(0x7f777c1430c0 legacy=0x7f777c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:26.379 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.379+0000 7f7779ffb640 1 -- 172.21.15.81:0/4048446953 >> v1:172.21.15.81:6789/0 conn(0x7f777c13f710 legacy=0x7f777c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:26.379 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.379+0000 7f7779ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:53:26.379 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.379+0000 7f7779ffb640 1 monclient: found mon.noname-c 2024-03-25T19:53:26.379 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.379+0000 7f7779ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:53:26.379 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.379+0000 7f7779ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.98:6789/0 2024-03-25T19:53:26.379 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.379+0000 7f7779ffb640 1 -- 172.21.15.81:0/4048446953 --> v1:172.21.15.98:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c0b44c0 con 0x7f777c0a98f0 2024-03-25T19:53:26.379 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.379+0000 7f7779ffb640 10 monclient: _finish_auth 0 2024-03-25T19:53:26.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.379+0000 7f7779ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:53:26.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.379+0000 7f7779ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:53:26.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.379+0000 7f7781663640 5 monclient: authenticate success, global_id 4729 2024-03-25T19:53:26.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.379+0000 7f7779ffb640 1 -- 172.21.15.81:0/4048446953 <== mon.1 v1:172.21.15.98:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f776c001690 con 0x7f777c0a98f0 2024-03-25T19:53:26.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.379+0000 7f7779ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:26.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.379+0000 7f7779ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:53:26.380 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.379+0000 7f7779ffb640 10 monclient: dump: 2024-03-25T19:53:26.380 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:53:26.380 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:26.380 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:26.380 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:26.380 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:26.381 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:26.381 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:53:26.381 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:53:26.381 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:53:26.381 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:26.381 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.379+0000 7f7781663640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:53:26.395 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.396+0000 7f7779ffb640 1 -- 172.21.15.81:0/4048446953 <== mon.1 v1:172.21.15.98:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f776c0017f0 con 0x7f777c0a98f0 2024-03-25T19:53:26.395 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.396+0000 7f7779ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:53:26.396 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.396+0000 7f7779ffb640 1 -- 172.21.15.81:0/4048446953 <== mon.1 v1:172.21.15.98:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f776c001c00 con 0x7f777c0a98f0 2024-03-25T19:53:26.396 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.396+0000 7f7779ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:26.396 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.396+0000 7f7779ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:53:26.396 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.396+0000 7f7779ffb640 10 monclient: dump: 2024-03-25T19:53:26.396 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:53:26.396 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:26.396 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:26.396 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:26.396 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:26.396 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:26.396 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:53:26.396 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:53:26.396 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:53:26.396 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:26.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.396+0000 7f7781663640 10 monclient: get_monmap_and_config success 2024-03-25T19:53:26.397 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.396+0000 7f7781663640 4 set_mon_vals no callback set 2024-03-25T19:53:26.399 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.397+0000 7f7781663640 10 monclient: shutdown 2024-03-25T19:53:26.399 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.397+0000 7f7781663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:53:26.399 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.397+0000 7f7781663640 1 -- 172.21.15.81:0/4048446953 >> v1:172.21.15.98:6789/0 conn(0x7f777c0a98f0 legacy=0x7f777c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:26.399 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.397+0000 7f7781663640 1 -- 172.21.15.81:0/4048446953 shutdown_connections 2024-03-25T19:53:26.399 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.397+0000 7f7781663640 1 -- 172.21.15.81:0/4048446953 >> 172.21.15.81:0/4048446953 conn(0x7f777c0b7ea0 msgr2=0x7f777c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:53:26.399 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.397+0000 7f7781663640 1 -- 172.21.15.81:0/4048446953 shutdown_connections 2024-03-25T19:53:26.399 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.397+0000 7f7781663640 1 -- 172.21.15.81:0/4048446953 wait complete. 2024-03-25T19:53:26.399 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.398+0000 7f7781663640 10 monclient: build_initial_monmap 2024-03-25T19:53:26.400 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.398+0000 7f7781663640 10 monclient: monmap: 2024-03-25T19:53:26.400 INFO:teuthology.misc.health.smithi081.stderr:epoch 0 2024-03-25T19:53:26.400 INFO:teuthology.misc.health.smithi081.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:53:26.400 INFO:teuthology.misc.health.smithi081.stderr:last_changed 0.000000 2024-03-25T19:53:26.400 INFO:teuthology.misc.health.smithi081.stderr:created 0.000000 2024-03-25T19:53:26.400 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 0 (unknown) 2024-03-25T19:53:26.400 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 1 2024-03-25T19:53:26.400 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.noname-a 2024-03-25T19:53:26.400 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.noname-b 2024-03-25T19:53:26.400 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.noname-c 2024-03-25T19:53:26.400 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:26.400 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.398+0000 7f7781663640 1 librados: starting msgr at 2024-03-25T19:53:26.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.398+0000 7f7781663640 1 librados: starting objecter 2024-03-25T19:53:26.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.398+0000 7f7781663640 1 Processor -- start 2024-03-25T19:53:26.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.398+0000 7f7781663640 1 -- start start 2024-03-25T19:53:26.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.398+0000 7f7781663640 1 librados: setting wanted keys 2024-03-25T19:53:26.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.398+0000 7f7781663640 1 librados: calling monclient init 2024-03-25T19:53:26.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.398+0000 7f7781663640 10 monclient: init 2024-03-25T19:53:26.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.399+0000 7f7781663640 10 monclient: _reopen_session rank -1 2024-03-25T19:53:26.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.399+0000 7f7781663640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:53:26.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.399+0000 7f7781663640 10 monclient(hunting): picked mon.noname-b con 0x7f777c0a98f0 addr v1:172.21.15.98:6789/0 2024-03-25T19:53:26.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.399+0000 7f7781663640 10 monclient(hunting): picked mon.noname-a con 0x7f777c13f710 addr v1:172.21.15.81:6789/0 2024-03-25T19:53:26.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.399+0000 7f7781663640 10 monclient(hunting): picked mon.noname-c con 0x7f777c1430c0 addr v1:172.21.15.81:6790/0 2024-03-25T19:53:26.401 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.399+0000 7f7781663640 1 -- --> v1:172.21.15.81:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f777c1db7d0 con 0x7f777c13f710 2024-03-25T19:53:26.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.399+0000 7f7781663640 1 -- --> v1:172.21.15.98:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f777c0b1420 con 0x7f777c0a98f0 2024-03-25T19:53:26.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.399+0000 7f7781663640 1 -- --> v1:172.21.15.81:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f777c0b15b0 con 0x7f777c1430c0 2024-03-25T19:53:26.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.399+0000 7f7781663640 10 monclient(hunting): _renew_subs 2024-03-25T19:53:26.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.399+0000 7f7781663640 10 monclient(hunting): authenticate will time out at 4641.481348s 2024-03-25T19:53:26.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.403+0000 7f777affd640 1 --1- >> v1:172.21.15.98:6789/0 conn(0x7f777c0a98f0 0x7f777c1daa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.98:6789/0 says I am v1:172.21.15.81:39136/0 (socket says 172.21.15.81:39136) 2024-03-25T19:53:26.402 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.403+0000 7f777affd640 1 -- 172.21.15.81:0/475410766 learned_addr learned my addr 172.21.15.81:0/475410766 (peer_addr_for_me v1:172.21.15.81:0/0) 2024-03-25T19:53:26.403 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.404+0000 7f77637fe640 10 client.?.objecter ms_handle_connect 0x7f777c0a98f0 2024-03-25T19:53:26.404 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.404+0000 7f77637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:26.410 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.411+0000 7f77637fe640 10 client.?.objecter ms_handle_connect 0x7f777c1430c0 2024-03-25T19:53:26.411 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.411+0000 7f77637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:26.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.417+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 <== mon.1 v1:172.21.15.98:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113436623 0 0) 0x7f777c0b1420 con 0x7f777c0a98f0 2024-03-25T19:53:26.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.417+0000 7f77637fe640 10 monclient(hunting): _negotiate 2024-03-25T19:53:26.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.417+0000 7f77637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:26.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.417+0000 7f77637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:26.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.417+0000 7f77637fe640 10 monclient(hunting): my global_id is 4732 2024-03-25T19:53:26.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.417+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 --> v1:172.21.15.98:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7768004eb0 con 0x7f777c0a98f0 2024-03-25T19:53:26.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.418+0000 7f77637fe640 10 client.?.objecter ms_handle_connect 0x7f777c13f710 2024-03-25T19:53:26.417 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.418+0000 7f77637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:26.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.423+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 <== mon.2 v1:172.21.15.81:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157905984 0 0) 0x7f777c0b15b0 con 0x7f777c1430c0 2024-03-25T19:53:26.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.423+0000 7f77637fe640 10 monclient(hunting): _negotiate 2024-03-25T19:53:26.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.423+0000 7f77637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:26.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.423+0000 7f77637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:26.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.423+0000 7f77637fe640 10 monclient(hunting): my global_id is 4715 2024-03-25T19:53:26.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.423+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 --> v1:172.21.15.81:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f777c0b1420 con 0x7f777c1430c0 2024-03-25T19:53:26.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.423+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 <== mon.0 v1:172.21.15.81:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125327924 0 0) 0x7f777c1db7d0 con 0x7f777c13f710 2024-03-25T19:53:26.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.423+0000 7f77637fe640 10 monclient(hunting): _negotiate 2024-03-25T19:53:26.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.423+0000 7f77637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:26.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.423+0000 7f77637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:26.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.423+0000 7f77637fe640 10 monclient(hunting): my global_id is 4758 2024-03-25T19:53:26.423 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.423+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 --> v1:172.21.15.81:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f777c0b15b0 con 0x7f777c13f710 2024-03-25T19:53:26.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.431+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 <== mon.0 v1:172.21.15.81:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1201008613 0 0) 0x7f777c0b15b0 con 0x7f777c13f710 2024-03-25T19:53:26.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.431+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 --> v1:172.21.15.81:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f777c1db7d0 con 0x7f777c13f710 2024-03-25T19:53:26.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.431+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 <== mon.0 v1:172.21.15.81:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f7770001830 con 0x7f777c13f710 2024-03-25T19:53:26.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.431+0000 7f77637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:26.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.431+0000 7f77637fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:53:26.431 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.431+0000 7f77637fe640 10 monclient(hunting): dump: 2024-03-25T19:53:26.431 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:53:26.432 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:26.432 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:26.432 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:26.432 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:26.432 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:26.432 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:53:26.432 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:53:26.432 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:53:26.432 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:26.432 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.431+0000 7f77637fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:53:26.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.439+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 <== mon.0 v1:172.21.15.81:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 544649097 0 0) 0x7f777c1db7d0 con 0x7f777c13f710 2024-03-25T19:53:26.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.439+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 >> v1:172.21.15.81:6790/0 conn(0x7f777c1430c0 legacy=0x7f777c1d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:26.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.439+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 >> v1:172.21.15.98:6789/0 conn(0x7f777c0a98f0 legacy=0x7f777c1daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:26.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.439+0000 7f77637fe640 10 monclient: _finish_hunting 0 2024-03-25T19:53:26.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.439+0000 7f77637fe640 1 monclient: found mon.a 2024-03-25T19:53:26.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.439+0000 7f77637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:53:26.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.439+0000 7f77637fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:53:26.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.439+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 --> v1:172.21.15.81:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c0b1740 con 0x7f777c13f710 2024-03-25T19:53:26.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.439+0000 7f77637fe640 10 monclient: _finish_auth 0 2024-03-25T19:53:26.440 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.439+0000 7f77637fe640 10 monclient: _check_auth_tickets 2024-03-25T19:53:26.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.439+0000 7f77637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:53:26.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.439+0000 7f7781663640 5 monclient: authenticate success, global_id 4758 2024-03-25T19:53:26.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.439+0000 7f7781663640 10 monclient: _renew_subs 2024-03-25T19:53:26.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.439+0000 7f7781663640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:53:26.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.439+0000 7f7781663640 1 -- 172.21.15.81:0/475410766 --> v1:172.21.15.81:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f777c1db1a0 con 0x7f777c13f710 2024-03-25T19:53:26.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.440+0000 7f7781663640 10 client.4758.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:53:26.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.440+0000 7f7781663640 10 monclient: _renew_subs 2024-03-25T19:53:26.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.440+0000 7f7781663640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:53:26.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.440+0000 7f7781663640 1 -- 172.21.15.81:0/475410766 --> v1:172.21.15.81:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f777c0b08f0 con 0x7f777c13f710 2024-03-25T19:53:26.441 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.440+0000 7f7781663640 1 librados: init done 2024-03-25T19:53:26.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.441+0000 7f7781663640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:26.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.441+0000 7f7781663640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:26.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.441+0000 7f7781663640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:53:26.442 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.441+0000 7f7781663640 1 -- 172.21.15.81:0/475410766 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777c0627b0 con 0x7f777c13f710 2024-03-25T19:53:26.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.764+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 <== mon.0 v1:172.21.15.81:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7770001d40 con 0x7f777c13f710 2024-03-25T19:53:26.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.764+0000 7f77637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:53:26.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.764+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 <== mon.0 v1:172.21.15.81:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2241924923 0 0) 0x7f7770003790 con 0x7f777c13f710 2024-03-25T19:53:26.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.764+0000 7f77637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:26.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.764+0000 7f77637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:53:26.764 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.764+0000 7f77637fe640 10 monclient: dump: 2024-03-25T19:53:26.764 INFO:teuthology.misc.health.smithi081.stderr:epoch 1 2024-03-25T19:53:26.764 INFO:teuthology.misc.health.smithi081.stderr:fsid 0b5243e3-4e87-434a-ac8b-84a1b1f413df 2024-03-25T19:53:26.764 INFO:teuthology.misc.health.smithi081.stderr:last_changed 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:26.765 INFO:teuthology.misc.health.smithi081.stderr:created 2024-03-25T19:41:04.424178+0000 2024-03-25T19:53:26.765 INFO:teuthology.misc.health.smithi081.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:26.765 INFO:teuthology.misc.health.smithi081.stderr:election_strategy: 3 2024-03-25T19:53:26.765 INFO:teuthology.misc.health.smithi081.stderr:0: v1:172.21.15.81:6789/0 mon.a 2024-03-25T19:53:26.765 INFO:teuthology.misc.health.smithi081.stderr:1: v1:172.21.15.98:6789/0 mon.b 2024-03-25T19:53:26.765 INFO:teuthology.misc.health.smithi081.stderr:2: v1:172.21.15.81:6790/0 mon.c 2024-03-25T19:53:26.765 INFO:teuthology.misc.health.smithi081.stderr: 2024-03-25T19:53:26.765 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.764+0000 7f77797fa640 4 set_mon_vals no callback set 2024-03-25T19:53:26.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.765+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 <== mon.0 v1:172.21.15.81:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2886397974 0 0) 0x7f77700039b0 con 0x7f777c13f710 2024-03-25T19:53:26.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.765+0000 7f777affd640 10 monclient: get_auth_request con 0x7f776806de20 auth_method 0 2024-03-25T19:53:26.766 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.766+0000 7f77637fe640 10 client.4758.objecter ms_handle_connect 0x7f776806de20 2024-03-25T19:53:26.770 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.770+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 <== mon.0 v1:172.21.15.81:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3447814714 0 0) 0x7f7770053500 con 0x7f777c13f710 2024-03-25T19:53:26.770 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.770+0000 7f77637fe640 10 client.4758.objecter ms_dispatch 0x7f777c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:53:26.770 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.770+0000 7f77637fe640 3 client.4758.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:53:26.770 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.770+0000 7f77637fe640 3 client.4758.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:53:26.770 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.770+0000 7f77637fe640 20 client.4758.objecter dump_active .. 0 homeless 2024-03-25T19:53:26.770 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.770+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 <== mon.0 v1:172.21.15.81:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f777004ee40 con 0x7f777c13f710 2024-03-25T19:53:26.770 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.770+0000 7f77637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:26.770 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:26.770+0000 7f77637fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:53:27.003 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.003+0000 7f7781663640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:27.003 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.003+0000 7f7781663640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:27.003 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.003+0000 7f7781663640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.81:6789/0 2024-03-25T19:53:27.003 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.003+0000 7f7781663640 1 -- 172.21.15.81:0/475410766 --> v1:172.21.15.81:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f777c0abb10 con 0x7f777c13f710 2024-03-25T19:53:27.032 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.032+0000 7f77637fe640 1 -- 172.21.15.81:0/475410766 <== mon.0 v1:172.21.15.81:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3235219182) 0x7f7770050f40 con 0x7f777c13f710 2024-03-25T19:53:27.032 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.032+0000 7f77637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:27.032 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.032+0000 7f77637fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:53:27.032 INFO:teuthology.misc.health.smithi081.stdout: 2024-03-25T19:53:27.032 INFO:teuthology.misc.health.smithi081.stdout:{"epoch":11,"fsid":"0b5243e3-4e87-434a-ac8b-84a1b1f413df","created":"2024-03-25T19:41:59.546919+0000","modified":"2024-03-25T19:52:03.690842+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:52:03.690842+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b3e439-45ad-438e-a3a0-a7eb0d3820c6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4838341b-32d7-4f3d-8251-78eacaa81b32","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ebc66313-ec95-428e-9fa9-559c14a21152","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"be8865f5-e87c-4abe-9bc2-c8e9aa7123bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"35265d96-239a-479d-92fb-2508747e7475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"bbbc4f31-6814-4509-a08a-d30bce723396","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36f755b4-5184-4bd5-8211-e880c2ae2551","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aed342a4-70bb-42a1-a26e-bbc15b6d0234","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:53:27.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.034+0000 7f7781663640 10 librados: watch_flush enter 2024-03-25T19:53:27.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.035+0000 7f7781663640 10 librados: watch_flush exit 2024-03-25T19:53:27.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.035+0000 7f7781663640 20 client.4758.objecter shutdown clearing up homeless session... 2024-03-25T19:53:27.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.035+0000 7f7781663640 10 client.4758.objecter successfully canceled tick 2024-03-25T19:53:27.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.035+0000 7f7781663640 1 -- 172.21.15.81:0/475410766 >> v1:172.21.15.98:6800/3980829775 conn(0x7f776806de20 legacy=0x7f77680702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:27.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.035+0000 7f7781663640 10 monclient: shutdown 2024-03-25T19:53:27.035 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.035+0000 7f7781663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:53:27.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.035+0000 7f7781663640 1 -- 172.21.15.81:0/475410766 >> v1:172.21.15.81:6789/0 conn(0x7f777c13f710 legacy=0x7f777c0b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:27.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.035+0000 7f7781663640 1 -- 172.21.15.81:0/475410766 shutdown_connections 2024-03-25T19:53:27.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.035+0000 7f7781663640 1 -- 172.21.15.81:0/475410766 >> 172.21.15.81:0/475410766 conn(0x7f777c0b7ea0 msgr2=0x7f777c13c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:53:27.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.035+0000 7f7781663640 1 -- 172.21.15.81:0/475410766 shutdown_connections 2024-03-25T19:53:27.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.035+0000 7f7781663640 1 -- 172.21.15.81:0/475410766 wait complete. 2024-03-25T19:53:27.036 INFO:teuthology.misc.health.smithi081.stderr:2024-03-25T19:53:27.035+0000 7f7781663640 1 librados: shutdown 2024-03-25T19:53:27.047 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:53:27.047 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:53:27.049 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:53:27.049 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:53:27.049 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:28.964 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:11:23.398 35594== 2024-03-25T19:53:28.965 INFO:tasks.ceph.osd.0.smithi081.stderr:==00:00:11:23.398 35594== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:29.000 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:11:51.642 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:29.004 INFO:tasks.ceph.osd.0.smithi081.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:29.045 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:51.680 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:29.046 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:51.681 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:29.047 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:11:51.688 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:29.052 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:11:51.694 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:29.059 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:11:51.701 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:29.077 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:11:51.712 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:29.099 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:11:51.741 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:29.105 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:11:51.747 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:33.152 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:33.152 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:53:33.153 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:53:33.153 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:53:33.153 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:35.026 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:11:29.464 35593== 2024-03-25T19:53:35.026 INFO:tasks.ceph.osd.1.smithi081.stderr:==00:00:11:29.464 35593== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:35.062 INFO:tasks.ceph.osd.1.smithi081.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:39.255 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:53:39.255 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:53:39.256 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:53:39.256 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:41.240 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:11:35.661 35595== 2024-03-25T19:53:41.241 INFO:tasks.ceph.osd.2.smithi081.stderr:==00:00:11:35.661 35595== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:41.281 INFO:tasks.ceph.osd.2.smithi081.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:45.357 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:45.358 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:53:45.358 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:53:45.358 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:53:45.358 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:47.301 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:11:41.719 35596== 2024-03-25T19:53:47.302 INFO:tasks.ceph.osd.3.smithi081.stderr:==00:00:11:41.719 35596== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:47.335 INFO:tasks.ceph.osd.3.smithi081.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:49.038 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:12:11.680 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:49.087 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:12:11.728 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:49.117 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:12:11.758 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:49.135 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:12:11.769 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:49.137 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:12:11.772 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:49.144 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:12:11.786 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:49.148 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:12:11.790 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:49.167 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:12:11.801 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:49.172 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:12:11.813 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:51.460 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:51.460 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi081 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:53:51.461 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:53:51.461 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:53:51.461 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:53.565 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:11:47.986 35242== 2024-03-25T19:53:53.565 INFO:tasks.ceph.osd.4.smithi098.stderr:==00:00:11:47.986 35242== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:53.603 INFO:tasks.ceph.osd.4.smithi098.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:57.562 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:57.563 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:53:57.563 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:53:57.563 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:53:57.564 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:59.449 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:11:53.886 35240== 2024-03-25T19:53:59.449 INFO:tasks.ceph.osd.5.smithi098.stderr:==00:00:11:53.886 35240== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:59.489 INFO:tasks.ceph.osd.5.smithi098.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:54:03.665 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:54:03.666 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:54:03.666 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:54:03.666 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:54:03.666 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:54:05.362 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:11:59.780 35243== 2024-03-25T19:54:05.362 INFO:tasks.ceph.osd.6.smithi098.stderr:==00:00:11:59.780 35243== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:54:05.398 INFO:tasks.ceph.osd.6.smithi098.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:54:09.132 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:12:31.774 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:54:09.139 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:12:31.781 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:54:09.153 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:12:31.795 35279== Warning: unimplemented fcntl command: 1036 2024-03-25T19:54:09.178 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:12:31.813 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:54:09.181 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:12:31.815 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:54:09.183 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:12:31.825 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:54:09.185 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:12:31.827 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:54:09.193 INFO:tasks.ceph.mon.b.smithi098.stderr:==00:00:12:31.827 34657== Warning: unimplemented fcntl command: 1036 2024-03-25T19:54:09.197 INFO:tasks.ceph.mon.c.smithi081.stderr:==00:00:12:31.839 35283== Warning: unimplemented fcntl command: 1036 2024-03-25T19:54:09.768 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:54:09.768 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:54:09.768 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:54:09.769 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:54:09.769 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:54:11.328 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:12:05.755 35241== 2024-03-25T19:54:11.329 INFO:tasks.ceph.osd.7.smithi098.stderr:==00:00:12:05.755 35241== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:54:11.358 INFO:tasks.ceph.osd.7.smithi098.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:54:15.870 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:54:15.871 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:54:15.871 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:54:15.871 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:54:15.871 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:54:15.872 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:54:15.896 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:54:15.897 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:54:15.897 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:54:15.898 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:54:15.899+0000 7fbdd6f99640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34655) UID: 0 2024-03-25T19:54:15.898 INFO:tasks.ceph.mgr.x.smithi098.stderr:2024-03-25T19:54:15.899+0000 7fbdd6f99640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T19:54:15.929 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:54:15.929 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:54:15.929 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:54:15.930 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:54:15.951 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-03-25T19:54:15.941+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35248) UID: 0 2024-03-25T19:54:15.952 INFO:tasks.ceph.mon.a.smithi081.stderr:2024-03-25T19:54:15.943+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T19:59:14.999 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:59:15.000 DEBUG:teuthology.orchestra.run.smithi081:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:59:15.045 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T19:59:15.046 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:59:15.119 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T19:59:15.120 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:59:15.185 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T19:59:15.186 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:59:15.253 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T19:59:15.253 DEBUG:teuthology.orchestra.run.smithi081:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:59:15.303 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T19:59:15.303 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:59:15.372 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T19:59:15.372 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:59:15.436 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T19:59:15.436 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:59:15.504 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T19:59:15.504 DEBUG:teuthology.orchestra.run.smithi098:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:59:15.561 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:59:15.562 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621334/data/mon.a.tgz 2024-03-25T19:59:15.563 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-03-25T19:59:15.577 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.fkuagEzbST 2024-03-25T19:59:15.577 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.fkuagEzbST 2024-03-25T19:59:15.674 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.fkuagEzbST 2024-03-25T19:59:15.756 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.fkuagEzbST is 133KB 2024-03-25T19:59:15.806 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.fkuagEzbST 2024-03-25T19:59:15.820 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621334/data/mon.c.tgz 2024-03-25T19:59:15.820 DEBUG:teuthology.orchestra.run.smithi081:> mktemp 2024-03-25T19:59:15.873 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.xBfTfHam04 2024-03-25T19:59:15.874 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.xBfTfHam04 2024-03-25T19:59:15.965 DEBUG:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.xBfTfHam04 2024-03-25T19:59:16.048 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.xBfTfHam04 is 166KB 2024-03-25T19:59:16.098 DEBUG:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.xBfTfHam04 2024-03-25T19:59:16.113 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621334/data/mon.b.tgz 2024-03-25T19:59:16.114 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-03-25T19:59:16.136 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.nh5aUzTxD8 2024-03-25T19:59:16.137 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.nh5aUzTxD8 2024-03-25T19:59:16.251 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.nh5aUzTxD8 2024-03-25T19:59:16.338 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.nh5aUzTxD8 is 156KB 2024-03-25T19:59:16.391 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.nh5aUzTxD8 2024-03-25T19:59:16.406 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:59:16.406 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:59:16.409 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:59:16.549 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 52, column 1 (, line 52) 2024-03-25T19:59:16.594 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 52, column 1 (, line 52) 2024-03-25T19:59:16.788 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-03-25T19:59:16.864 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 52, column 1 (, line 52) 2024-03-25T19:59:17.035 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:59:17.037 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621334/remote/smithi081/crash 2024-03-25T19:59:17.038 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:59:17.068 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621334/remote/smithi098/crash 2024-03-25T19:59:17.069 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:59:17.096 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:59:17.096 DEBUG:teuthology.orchestra.run.smithi081:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:59:17.109 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:59:17.183 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:59:17.183 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35014.log 2024-03-25T19:59:17.183 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:59:17.183 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35014.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:59:17.183 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/valgrind/osd.7.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35014.log.gz 2024-03-25T19:59:17.183 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:59:17.183 INFO:teuthology.orchestra.run.smithi098.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:59:17.183 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:59:17.183 INFO:teuthology.orchestra.run.smithi098.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T19:59:17.183 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:59:17.184 INFO:teuthology.orchestra.run.smithi098.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:59:17.184 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34810.log 2024-03-25T19:59:17.184 INFO:teuthology.orchestra.run.smithi098.stderr: 55.8% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:59:17.184 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34850.log 2024-03-25T19:59:17.184 INFO:teuthology.orchestra.run.smithi098.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:59:17.184 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35101.log 2024-03-25T19:59:17.184 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34930.log 2024-03-25T19:59:17.184 INFO:teuthology.orchestra.run.smithi098.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34810.log.gz 2024-03-25T19:59:17.184 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34970.log 2024-03-25T19:59:17.184 INFO:teuthology.orchestra.run.smithi098.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34850.log.gz 2024-03-25T19:59:17.184 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34930.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:59:17.184 INFO:teuthology.orchestra.run.smithi098.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.35101.log.gz 2024-03-25T19:59:17.184 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34970.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34930.log.gz 2024-03-25T19:59:17.184 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:59:17.185 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34890.log 2024-03-25T19:59:17.185 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34970.log.gz 2024-03-25T19:59:17.185 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35061.log 2024-03-25T19:59:17.185 INFO:teuthology.orchestra.run.smithi098.stderr: 88.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:59:17.185 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.34890.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:59:17.185 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.admin.35061.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:59:17.185 INFO:teuthology.orchestra.run.smithi098.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34890.log.gz 2024-03-25T19:59:17.185 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.7.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35061.log.gz 2024-03-25T19:59:17.185 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:59:17.185 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:59:17.185 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-osd.5.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T19:59:17.185 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:59:17.196 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40144.log 2024-03-25T19:59:17.196 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:59:17.196 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41158.log 2024-03-25T19:59:17.196 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40144.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41406.log 2024-03-25T19:59:17.196 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40144.log.gz 2024-03-25T19:59:17.196 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42265.log 2024-03-25T19:59:17.196 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41314.log 2024-03-25T19:59:17.196 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41158.log.gz 2024-03-25T19:59:17.196 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40417.log 2024-03-25T19:59:17.196 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41406.log.gz 2024-03-25T19:59:17.196 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41314.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:59:17.197 INFO:teuthology.orchestra.run.smithi081.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42265.log.gz 2024-03-25T19:59:17.197 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40417.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:59:17.197 INFO:teuthology.orchestra.run.smithi081.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41314.log.gz 2024-03-25T19:59:17.197 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose --/var/log/ceph/valgrind/mon.c.log: /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:59:17.197 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40417.log.gz 2024-03-25T19:59:17.197 INFO:teuthology.orchestra.run.smithi081.stderr: 55.9% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T19:59:17.197 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:59:17.197 INFO:teuthology.orchestra.run.smithi081.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:59:17.197 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:59:17.197 INFO:teuthology.orchestra.run.smithi081.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:59:17.197 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:59:17.197 INFO:teuthology.orchestra.run.smithi081.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:59:17.197 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29439.log 2024-03-25T19:59:17.197 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/valgrind/mon.a.log: 55.7% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:59:17.197 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40846.log 2024-03-25T19:59:17.198 INFO:teuthology.orchestra.run.smithi081.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:59:17.198 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.tmp-client.admin.29439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41445.log 2024-03-25T19:59:17.198 INFO:teuthology.orchestra.run.smithi081.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29439.log.gz 2024-03-25T19:59:17.198 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40339.log 2024-03-25T19:59:17.198 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42306.log 2024-03-25T19:59:17.198 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40846.log.gz 2024-03-25T19:59:17.198 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40339.log: 86.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.41445.log.gz 2024-03-25T19:59:17.198 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42187.log 2024-03-25T19:59:17.198 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41602.log 2024-03-25T19:59:17.198 INFO:teuthology.orchestra.run.smithi081.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40339.log.gz 2024-03-25T19:59:17.198 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35750.log 2024-03-25T19:59:17.198 INFO:teuthology.orchestra.run.smithi081.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42306.log.gz 2024-03-25T19:59:17.198 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.42187.log.gz 2024-03-25T19:59:17.198 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40963.log 2024-03-25T19:59:17.199 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41602.log: /var/log/ceph/ceph-client.admin.35750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41119.log 2024-03-25T19:59:17.199 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40963.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41602.log.gz 2024-03-25T19:59:17.199 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40105.log 2024-03-25T19:59:17.199 INFO:teuthology.orchestra.run.smithi081.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35750.log.gz 2024-03-25T19:59:17.199 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41119.log: gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40963.log.gz 2024-03-25T19:59:17.199 INFO:teuthology.orchestra.run.smithi081.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39675.log 2024-03-25T19:59:17.199 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41641.log 2024-03-25T19:59:17.199 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41119.log.gz 2024-03-25T19:59:17.199 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39675.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:59:17.199 INFO:teuthology.orchestra.run.smithi081.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40105.log.gz 2024-03-25T19:59:17.199 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39636.log 2024-03-25T19:59:17.199 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39675.log.gz 2024-03-25T19:59:17.199 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39910.log 2024-03-25T19:59:17.199 INFO:teuthology.orchestra.run.smithi081.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41641.log.gz 2024-03-25T19:59:17.200 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41002.log 2024-03-25T19:59:17.200 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39910.log: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:59:17.200 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39363.log 2024-03-25T19:59:17.200 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39636.log.gz 2024-03-25T19:59:17.200 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41002.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39910.log.gz 2024-03-25T19:59:17.200 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42109.log 2024-03-25T19:59:17.200 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41914.log 2024-03-25T19:59:17.200 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39363.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41002.log.gz 2024-03-25T19:59:17.200 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38915.log 2024-03-25T19:59:17.200 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41914.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39363.log.gz 2024-03-25T19:59:17.200 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42070.log 2024-03-25T19:59:17.200 INFO:teuthology.orchestra.run.smithi081.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42109.log.gz 2024-03-25T19:59:17.200 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.38915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40261.log 2024-03-25T19:59:17.200 INFO:teuthology.orchestra.run.smithi081.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41914.log.gz 2024-03-25T19:59:17.201 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41080.log 2024-03-25T19:59:17.201 INFO:teuthology.orchestra.run.smithi081.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38915.log.gz 2024-03-25T19:59:17.201 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40300.log 2024-03-25T19:59:17.201 INFO:teuthology.orchestra.run.smithi081.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42070.log.gz 2024-03-25T19:59:17.201 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41836.log 2024-03-25T19:59:17.201 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40261.log.gz 2024-03-25T19:59:17.201 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40300.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41080.log.gz 2024-03-25T19:59:17.201 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41524.log 2024-03-25T19:59:17.201 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40183.log 2024-03-25T19:59:17.201 INFO:teuthology.orchestra.run.smithi081.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40300.log.gz 2024-03-25T19:59:17.201 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41524.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41836.log.gz 2024-03-25T19:59:17.201 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41992.log 2024-03-25T19:59:17.201 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39245.log 2024-03-25T19:59:17.201 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40183.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41524.log.gz 2024-03-25T19:59:17.201 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37630.log 2024-03-25T19:59:17.202 INFO:teuthology.orchestra.run.smithi081.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40183.log.gz 2024-03-25T19:59:17.202 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40027.log 2024-03-25T19:59:17.202 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41992.log.gz 2024-03-25T19:59:17.202 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.37630.log: gzip 86.9% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.39245.log.gz -- 2024-03-25T19:59:17.202 INFO:teuthology.orchestra.run.smithi081.stderr: /var/log/ceph/ceph-client.admin.40924.log 2024-03-25T19:59:17.202 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40027.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.37630.log.gz 2024-03-25T19:59:17.202 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40768.log 2024-03-25T19:59:17.202 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39753.log 2024-03-25T19:59:17.202 INFO:teuthology.orchestra.run.smithi081.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40027.log.gz 2024-03-25T19:59:17.202 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39714.log 2024-03-25T19:59:17.202 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40924.log.gz 2024-03-25T19:59:17.202 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40768.log.gz 2024-03-25T19:59:17.202 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39753.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:59:17.202 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35619.log 2024-03-25T19:59:17.203 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39753.log.gz 2024-03-25T19:59:17.203 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41953.log 2024-03-25T19:59:17.203 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39714.log.gz 2024-03-25T19:59:17.203 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40222.log 2024-03-25T19:59:17.203 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39480.log 2024-03-25T19:59:17.203 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.35619.log.gz 2024-03-25T19:59:17.203 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40222.log: 86.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42345.log 2024-03-25T19:59:17.203 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41953.log.gz 2024-03-25T19:59:17.203 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41719.log 2024-03-25T19:59:17.203 INFO:teuthology.orchestra.run.smithi081.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40222.log.gz 2024-03-25T19:59:17.203 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39832.log 2024-03-25T19:59:17.203 INFO:teuthology.orchestra.run.smithi081.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39480.log.gz 2024-03-25T19:59:17.203 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41719.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42345.log.gz 2024-03-25T19:59:17.204 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41236.log 2024-03-25T19:59:17.204 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39519.log 2024-03-25T19:59:17.204 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39832.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41719.log.gz 2024-03-25T19:59:17.204 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41680.log 2024-03-25T19:59:17.204 INFO:teuthology.orchestra.run.smithi081.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39832.log.gz/var/log/ceph/ceph-client.admin.39519.log: 2024-03-25T19:59:17.204 INFO:teuthology.orchestra.run.smithi081.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41236.log.gz 2024-03-25T19:59:17.204 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39402.log 2024-03-25T19:59:17.204 INFO:teuthology.orchestra.run.smithi081.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39519.log.gz 2024-03-25T19:59:17.204 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40807.log 2024-03-25T19:59:17.204 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41353.log 2024-03-25T19:59:17.204 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39402.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41680.log.gz 2024-03-25T19:59:17.204 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41275.log 2024-03-25T19:59:17.204 INFO:teuthology.orchestra.run.smithi081.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39402.log.gz 2024-03-25T19:59:17.204 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41353.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40807.log.gz 2024-03-25T19:59:17.205 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41485.log 2024-03-25T19:59:17.205 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41353.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39206.log 2024-03-25T19:59:17.205 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T19:59:17.205 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41275.log: /var/log/ceph/ceph-client.admin.41485.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:59:17.205 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39206.log: 86.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40495.log 2024-03-25T19:59:17.205 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41275.log.gz 2024-03-25T19:59:17.205 INFO:teuthology.orchestra.run.smithi081.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41485.log.gz 2024-03-25T19:59:17.205 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41875.log 2024-03-25T19:59:17.205 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39206.log.gz 2024-03-25T19:59:17.205 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40729.log 2024-03-25T19:59:17.205 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40495.log: /var/log/ceph/ceph-client.admin.41875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35592.log 2024-03-25T19:59:17.205 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40495.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41563.log 2024-03-25T19:59:17.205 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T19:59:17.205 INFO:teuthology.orchestra.run.smithi081.stderr: 86.5%/var/log/ceph/ceph-client.admin.40729.log: -- replaced with /var/log/ceph/ceph-client.admin.41875.log.gz 2024-03-25T19:59:17.205 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41758.log 2024-03-25T19:59:17.206 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41563.log: 86.6% 86.3% -- replaced with /var/log/ceph/ceph-client.admin.35592.log.gz 2024-03-25T19:59:17.206 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40729.log.gz 2024-03-25T19:59:17.206 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42031.log 2024-03-25T19:59:17.206 INFO:teuthology.orchestra.run.smithi081.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41563.log.gz 2024-03-25T19:59:17.206 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42423.log 2024-03-25T19:59:17.206 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41758.log: /var/log/ceph/ceph-client.admin.42031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40378.log 2024-03-25T19:59:17.206 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39441.log 2024-03-25T19:59:17.206 INFO:teuthology.orchestra.run.smithi081.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42031.log.gz 2024-03-25T19:59:17.206 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40378.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41758.log.gz 2024-03-25T19:59:17.206 INFO:teuthology.orchestra.run.smithi081.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.42423.log.gz 2024-03-25T19:59:17.206 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42226.log 2024-03-25T19:59:17.206 INFO:teuthology.orchestra.run.smithi081.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40378.log.gz 2024-03-25T19:59:17.206 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39988.log 2024-03-25T19:59:17.206 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39441.log: /var/log/ceph/ceph-client.admin.42226.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:59:17.207 INFO:teuthology.orchestra.run.smithi081.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39441.log.gz 2024-03-25T19:59:17.207 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39988.log: 86.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40651.log 2024-03-25T19:59:17.207 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42226.log.gz 2024-03-25T19:59:17.207 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mgr.y.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39988.log.gz 2024-03-25T19:59:17.207 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42148.log 2024-03-25T19:59:17.207 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36634.log 2024-03-25T19:59:17.207 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40651.log: /var/log/ceph/ceph-client.admin.42148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35290.log 2024-03-25T19:59:17.207 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.36634.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40651.log.gz 2024-03-25T19:59:17.207 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42148.log.gz 2024-03-25T19:59:17.207 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39126.log 2024-03-25T19:59:17.207 INFO:teuthology.orchestra.run.smithi081.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.36634.log.gz 2024-03-25T19:59:17.207 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:59:17.207 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.35290.log: /var/log/ceph/ceph-client.admin.39126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40534.log 2024-03-25T19:59:17.207 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.0.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39126.log.gz 2024-03-25T19:59:17.208 INFO:teuthology.orchestra.run.smithi081.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.35290.log.gz 2024-03-25T19:59:17.208 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:59:17.208 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:59:17.208 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40534.log: /var/log/ceph/ceph.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40534.log.gz 2024-03-25T19:59:17.208 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41197.log 2024-03-25T19:59:17.208 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-osd.2.log: 92.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:59:17.208 INFO:teuthology.orchestra.run.smithi081.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T19:59:17.208 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40573.log 2024-03-25T19:59:17.208 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39597.log 2024-03-25T19:59:17.208 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41197.log.gz 2024-03-25T19:59:17.208 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39165.log 2024-03-25T19:59:17.209 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39597.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40573.log.gz 2024-03-25T19:59:17.209 INFO:teuthology.orchestra.run.smithi081.stderr: 85.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39558.log 2024-03-25T19:59:17.209 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39597.log.gz 2024-03-25T19:59:17.209 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42384.log 2024-03-25T19:59:17.209 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39558.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39165.log.gz 2024-03-25T19:59:17.209 INFO:teuthology.orchestra.run.smithi081.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39558.log.gz 2024-03-25T19:59:17.209 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39324.log 2024-03-25T19:59:17.209 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.42384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39793.log 2024-03-25T19:59:17.209 INFO:teuthology.orchestra.run.smithi081.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42384.log.gz 2024-03-25T19:59:17.209 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40066.log 2024-03-25T19:59:17.209 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39793.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39324.log.gz 2024-03-25T19:59:17.209 INFO:teuthology.orchestra.run.smithi081.stderr: 86.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39949.log 2024-03-25T19:59:17.209 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39793.log.gz 2024-03-25T19:59:17.209 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41797.log 2024-03-25T19:59:17.210 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40066.log: /var/log/ceph/ceph-client.admin.39949.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40066.log.gz 2024-03-25T19:59:17.210 INFO:teuthology.orchestra.run.smithi081.stderr: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40612.log 2024-03-25T19:59:17.210 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39949.log.gz 2024-03-25T19:59:17.210 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39871.log 2024-03-25T19:59:17.210 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40612.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41797.log.gz 2024-03-25T19:59:17.210 INFO:teuthology.orchestra.run.smithi081.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40612.log.gz 2024-03-25T19:59:17.210 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39285.log 2024-03-25T19:59:17.210 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41041.log 2024-03-25T19:59:17.210 INFO:teuthology.orchestra.run.smithi081.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39871.log.gz 2024-03-25T19:59:17.210 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.39285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40690.log 2024-03-25T19:59:17.210 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.41041.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39285.log.gz 2024-03-25T19:59:17.210 INFO:teuthology.orchestra.run.smithi081.stderr: 85.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40456.log 2024-03-25T19:59:17.210 INFO:teuthology.orchestra.run.smithi081.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41041.log.gz 2024-03-25T19:59:17.210 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40690.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:59:17.211 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-client.admin.40456.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40690.log.gz 2024-03-25T19:59:17.211 INFO:teuthology.orchestra.run.smithi081.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40456.log.gz 2024-03-25T19:59:17.211 INFO:teuthology.orchestra.run.smithi081.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40885.log 2024-03-25T19:59:17.211 INFO:teuthology.orchestra.run.smithi081.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.40885.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40885.log.gz 2024-03-25T19:59:17.224 INFO:teuthology.orchestra.run.smithi098.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:59:17.973 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:17:40.615 35279== 2024-03-25T19:59:17.973 INFO:tasks.ceph.mon.a.smithi081.stderr:==00:00:17:40.615 35279== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:59:17.990 INFO:tasks.ceph.mon.a.smithi081.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:59:18.104 INFO:teuthology.orchestra.run.smithi098.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:59:18.110 INFO:teuthology.orchestra.run.smithi098.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:59:18.212 INFO:teuthology.orchestra.run.smithi081.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:59:18.234 INFO:teuthology.orchestra.run.smithi081.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:59:18.271 INFO:teuthology.orchestra.run.smithi081.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:59:18.370 INFO:teuthology.orchestra.run.smithi098.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:59:18.385 INFO:teuthology.orchestra.run.smithi098.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:59:18.385 INFO:teuthology.orchestra.run.smithi081.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:59:22.194 INFO:teuthology.orchestra.run.smithi081.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2024-03-25T19:59:22.194 INFO:teuthology.orchestra.run.smithi081.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:59:22.316 INFO:teuthology.orchestra.run.smithi098.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2024-03-25T19:59:22.317 INFO:teuthology.orchestra.run.smithi098.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:59:22.318 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-25T19:59:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m5.163s 2024-03-25T19:59:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:user 0m9.239s 2024-03-25T19:59:22.318 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m0.263s 2024-03-25T19:59:22.510 INFO:teuthology.orchestra.run.smithi081.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping 2024-03-25T19:59:22.573 INFO:teuthology.orchestra.run.smithi081.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:59:22.574 INFO:teuthology.orchestra.run.smithi081.stderr: 2024-03-25T19:59:22.575 INFO:teuthology.orchestra.run.smithi081.stderr:real 0m5.454s 2024-03-25T19:59:22.575 INFO:teuthology.orchestra.run.smithi081.stderr:user 0m14.542s 2024-03-25T19:59:22.575 INFO:teuthology.orchestra.run.smithi081.stderr:sys 0m0.502s 2024-03-25T19:59:22.575 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:59:22.575 DEBUG:teuthology.misc:Transferring archived files from smithi081:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621334/remote/smithi081/log 2024-03-25T19:59:22.576 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:59:25.085 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621334/remote/smithi098/log 2024-03-25T19:59:25.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:59:26.692 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-03-25T19:59:26.917 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2c158f26361a409e930ca2f4d6bd2358 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-03-25T19:59:26.923 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:59:26.933 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:59:26.970 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:59:26.970 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:59:26.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:59:27.069 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:59:27.078 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:59:27.078 DEBUG:teuthology.orchestra.run.smithi081:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:59:27.081 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:59:27.095 INFO:teuthology.orchestra.run.smithi081.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:59:27.098 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:59:27.101 INFO:teuthology.orchestra.run.smithi081.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:59:27.101 INFO:teuthology.orchestra.run.smithi081.stdout:=============================================================================== 2024-03-25T19:59:27.101 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 -561us[ -553us] +/- 68ms 2024-03-25T19:59:27.101 INFO:teuthology.orchestra.run.smithi081.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -111us[ -103us] +/- 70ms 2024-03-25T19:59:27.102 INFO:teuthology.orchestra.run.smithi081.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +759us[ +767us] +/- 73ms 2024-03-25T19:59:27.106 INFO:teuthology.orchestra.run.smithi081.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:59:27.108 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:59:27.108 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-25T19:59:27.108 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 49 -500us[ -500us] +/- 68ms 2024-03-25T19:59:27.108 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 -72us[ -26us] +/- 70ms 2024-03-25T19:59:27.108 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +801us[ +801us] +/- 73ms 2024-03-25T19:59:27.108 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:59:27.109 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:59:27.117 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:59:27.118 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:59:27.127 DEBUG:teuthology.orchestra.run.smithi081:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:59:27.211 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:59:27.282 DEBUG:teuthology.orchestra.run.smithi081:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:59:27.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:59:27.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:59:27.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:59:27.345 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:59:27.356 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:59:27.366 INFO:teuthology.task.internal:Duration was 4647.225272 seconds 2024-03-25T19:59:27.366 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:59:27.375 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:59:27.375 DEBUG:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:59:27.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:59:27.419 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:59:27.430 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:59:27.813 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:59:27.814 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi081.front.sepia.ceph.com 2024-03-25T19:59:27.814 DEBUG:teuthology.orchestra.run.smithi081:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:59:27.834 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-03-25T19:59:27.834 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:59:27.862 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:59:27.862 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:59:27.877 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:59:27.931 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:59:27.932 DEBUG:teuthology.orchestra.run.smithi081:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:59:27.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:59:28.074 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:59:28.084 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:59:28.084 DEBUG:teuthology.orchestra.run.smithi081:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:59:28.111 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:59:28.138 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:59:28.148 DEBUG:teuthology.orchestra.run.smithi081:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:59:28.154 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:59:28.177 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = core 2024-03-25T19:59:28.204 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-03-25T19:59:28.224 DEBUG:teuthology.orchestra.run.smithi081:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:59:28.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:59:28.252 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:59:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:59:28.279 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:59:28.290 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:59:28.290 DEBUG:teuthology.misc:Transferring archived files from smithi081:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621334/remote/smithi081 2024-03-25T19:59:28.291 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:59:28.328 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621334/remote/smithi098 2024-03-25T19:59:28.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:59:28.365 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:59:28.366 DEBUG:teuthology.orchestra.run.smithi081:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:59:28.368 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:59:28.415 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:59:28.426 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:59:28.427 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:59:28.436 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:59:28.436 DEBUG:teuthology.orchestra.run.smithi081:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:59:28.438 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:59:28.450 INFO:teuthology.orchestra.run.smithi081.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:59 /home/ubuntu/cephtest 2024-03-25T19:59:28.469 INFO:teuthology.orchestra.run.smithi098.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:59 /home/ubuntu/cephtest 2024-03-25T19:59:28.471 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:59:28.481 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:59:28.500 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:59:28.527 DEBUG:teuthology.nuke:shortname: smithi081 2024-03-25T19:59:28.528 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:59:28.553 DEBUG:teuthology.nuke:shortname: smithi098 2024-03-25T19:59:28.553 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:59:28.557 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi081.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621334', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:32:26.971765', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCN07iZlXHTVUpCzBERZtT0MJipiUFcqZGMF7B9u5DuDBQr05X96yQDfh6RwCfxsDrBc/m6JwKQt4dYEYGHw30c='} 2024-03-25T19:59:28.577 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621334', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:32:26.970161', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAwY3yfX1UoEShtlLxQrA1uQ4Cv4BCM43c1vEnjfLjFwEVlcfA35mwkmImJeqzHFIeBGPMSMzTHJBoPWDpViQLY='} 2024-03-25T19:59:28.605 INFO:teuthology.orchestra.console.smithi081:Power off 2024-03-25T19:59:28.605 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:59:28.629 DEBUG:teuthology.orchestra.console.smithi081:power off output: Chassis Power Control: Down/Off 2024-03-25T19:59:28.629 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:59:28.648 INFO:teuthology.orchestra.console.smithi098:Power off 2024-03-25T19:59:28.649 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:59:28.659 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2024-03-25T19:59:28.672 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2024-03-25T19:59:28.672 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:59:28.697 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-25T19:59:32.661 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:59:32.698 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:59:32.773 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is on 2024-03-25T19:59:32.811 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-25T19:59:36.778 DEBUG:teuthology.orchestra.console.smithi081:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:59:36.812 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:59:36.890 DEBUG:teuthology.orchestra.console.smithi081:check power output: Chassis Power is off 2024-03-25T19:59:36.923 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-03-25T19:59:36.991 INFO:teuthology.orchestra.console.smithi081:Power off completed 2024-03-25T19:59:37.024 INFO:teuthology.orchestra.console.smithi098:Power off completed 2024-03-25T19:59:37.129 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 4647.225271940231 failure_reason: 'valgrind error: Leak_PossiblyLost calloc __trans_list_add ' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=2c158f26361a409e930ca2f4d6bd2358 status: fail success: false 2024-03-25T19:59:37.129 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:59:37.210 INFO:teuthology.run:FAIL